Usability: Don’t Listen, Just Watch

 When I’ve been undertaking usability testing, no matter how many times the facilitators tried to tell the participants they were evaluating the product and not them, the participants still got the sense they were the ones being “tested.” Their conversation and feedback are very much centered on their own abilities, not on the product. In general, this sense instills instant anxiety and is very difficult to get past, even if you are a master of dealing with nerves. It is a very rare occurrence for individuals to not be anxious when they feel as though they are being tested.

Granted, this doesn’t happen all the time. When engaged, professional user research participants will tell you exactly what they think of the product as they understand the importance of such information. However, how often do projects employ professional user-research participants? I’ve only seen one in my career, which spans over hundreds of projects. More often, projects employ professional testers, but their focus is generally “functional” testing and usability quickly becomes an afterthought—if at all.

My Anxious First Time Participating
Before running evaluations myself and undertaking usability testing as a professional, I experienced a usability evaluation as a participant.

It was an internal product evaluation, and I was new to the company. The test took place in a “usability lab” that had one-sided glass adjoining the observation room. There were three or four cameras recording me, and I knew there were also several people in the observation room as I had seen them all moving around before we started. I was a nervous wreck! It was like I was auditioning for some type of acting role. I do believe that it was more distressing for me than public speaking, and at the time that was one of my greatest fears.

I ran through the various scenarios as directed; however, to this day I question the value of any observations they may have made and any information they gained through feedback from me. My nerves impacted my reactions and product navigation, of that I have no doubt.

The facilitators offered the usual assurances, and told me that there was no need to be nervous; however nerves are not something that simply vanish on request. 

What Could They Have Done?
The best way forward in that scenario, from my personal perspective, would have been for one of the observers to sit next to me at a computer in a location that was familiar to me (my usual workstation) and taken simple notes as I moved through the scenarios; an approach that has become more common in UX design more recently.

I would not have been anywhere near as nervous and my reactions would have been less tainted. Yes, they would not have had their video recording, but as I mentioned, this footage is of no real use anyway. My emotional responses were too impacted by my nerves. It was not a total loss; I do recall discussing certain points of confusion. However, would those points have been as confusing if I was not so nervous and therefore clearer minded?  

After that event, it took years to build up the courage to participate in more evaluations. The only reason I did was due to the product being far more familiar to me by that time. It’s important to gauge how the participants are feeling before, during, and after an evaluation. Usability is subjective, and feelings impact decisions. As feelings change, so to do the associated decisions.  

No Feedback at All?
There are also times when you can’t worry about what the participants are saying, as they won’t say anything at all. When it comes to usability testing, there are no stupid questions. Make sure your participants are aware of that fact, and don’t be afraid to ask questions in order to prompt open responses from them. 

The Participant’s Point of View
Imagine you’re a participant in a usability evaluation of some description, sitting in a room in front of a computer with two, maybe more, people watching your every move; not unlike my first experience. This would be somewhat nerve-wracking if you were an expert user of the product, let alone not knowing it well (which most participants won’t). The potential to feel very inadequate as a user is quite high, and therefore questions would be easy to avoid asking.

How many times have you stopped yourself from asking a question as you were afraid of looking stupid? Earlier in my career I was guilty of this often. I was concerned of what the people around me would think if the question I asked had an obvious answer that I had not seen. 

As a software tester I have learned that questions are what we do; they are the core of our profession, so I have left that problem far behind. For many others though, particularly when faced with an anxious situation (i.e. participating in a usability evaluation), this is an all too common scenario. 

Someone once said to me, "If you don't fail, you don't learn." Well, many people have said that to me. Essentially, it’s the driving philosophy behind agile development and user-centered design. It's one of those statements that appears to be very logical, but is not so easy to put into practice. The fear of failure is what drives the majority of our anxiety in the corporate world. If you weren't afraid of messing up your speech to one-hundred-plus people, would you still be nervous? I think not, well not as much anyway.

Overcoming the “Stupid Question” Fear
What worked for me was starting small. Taking advantage of times where failure didn’t matter a great deal; meetings with small numbers of attendees, delivering training to small groups, etc. The lesson from this is to make the participants fear of failure as small as possible. As you're demonstrating the product to them or preparing them for the evaluation, perhaps mention some of your own inadequacies in relation to the product; but of course be very careful not to “lead the witness;” you need their feedback, not yours through their words. This may help their fear move to a more manageable level.    

Balancing the need to observe with not making participants nervous to the point of being useless can be complex to achieve. It will of course depend on the participant and their previous experiences as to how much you need to adjust your approach, but being armed with some easy-to-deploy and flexible techniques will put you in a good starting position. Here are a couple of mine that have proven valuable in the past

• A comfortable environment. Quite often this is the participant’s usual work space. They are surrounded by their work mates, they have their family and pet photos on their desk and you’re simply a visitor 'pulling up a chair'. Also, the need for filming cannot be a blocker to this anymore as setting up a smartphone to lean against the monitor is an easy and non-intrusive form of video capture.
• A soft introduction to the product. You need to be careful with this. The last thing you want to end up doing is providing a training session and therefore negating the evaluation almost completely. An introduction is common practice for an evaluation; however, there is an observation opportunity that is often overlooked (pun intended). Make the participant aware that it’s an introduction, which will help them be more comfortable as they will assume you’ll be doing the work, and while introducing the product make sure you observe them at the same time. You can gleam some very valuable information just from this first period of the evaluation. “Why did you have to do that?” A powerful question from the participant; and you’re only two minutes in.
• Understand their background. Talking to the participants about their experience is a valuable conversation. Asking them about their specific role in the company is great, but I also like to take it to a more personal level. Sharing a conversation about a favorite show or book adds an element of harmony to the situation. For those participants that are very much “work only” and are not willing to discuss personal details, you can be sneaky and build it into the evaluation. I have previously advised a participant that I needed to know some of these details so that we could more accurately assess the demographic of the user base. When she answered how many kids she had it was easy for me to ask their age and then to bring my child into the conversation, and away we went talking about schools and sport. The benefit to understanding their background is two-fold. First, you get to know what experience they have and how this may impact their use of the product. Second, it will assist in building a rapport with the participant, helping to ease those very important comfort levels. 

Getting participants to a state of mind where they are comfortable to honestly raise whatever issues they come across, including emotional responses, is the ultimate goal. If this does not work, observation may be the only avenue you have, so it’s important to keep your eyes open, and don’t always believe the participant when they say the product is easy to use.

Do you have any tips on how to ease the comfort levels of participants or examples of when observation was your only hope? Please let us know in the comments section below.