Key principles on usability test methodology

The key principles of the methodology of usability tests are based on the conventional scientific lab-testing approaches. You formulate an hypothesis regarding an issue that needs fixing, and then you analyze the causes and effects as well as their relation with each other in order to draw conclusions.

The attitude to adopt during a usability test

Before running a usability test and benefitting from its results, it is first necessary to understand creator does not equal the user. To ensure this, NN Group has crafted a list of recommendations, in the collection “230 Tips and Tricks for Better Usability Testing”, whose goals are to help craft and run the most efficient usability tests possible. Here are the nine recommendations that we consider essential:

1. Stay humble. This sums up in two words what your state of mind needs to be throughout the whole test.

2. Make sure your product's functionalities and their ease of use are optimal. Do not hesitate to involve your teams in this consideration process.

3. Regularly check the standard of your working process.

4. Find ways to maintain or enhance your skills.

5. Keep track of the existing publications in your area of expertise.

6. Avoid what has to do with personal opinion and, as much as possible, base your approach on hard data.

7. Avoid making developing decisions based only on the users' opinions. “Don't tell me, show me!” should be the formula for any usability test.

8. Do not test your own creation. No one is impartial enough to efficiently appraise criticism on what they have created. It is however preferable to test your own creation than to not test at all, if it is indeed the only option available to you.

9. It is entirely possible to have a previous participant test the product again once modifications have been made. This can even be particularly beneficial since the user already knows the product, and will thus be able to provide more in depth remarks. The results might however be biased, seeing as this will not in fact be the first time that the participant handles the product, and might want to prove his or her previous remarks were accurate.

Interaction with the participant:

If the creator has no choice but to endorse the role of moderator during the test, it is essential that he or she has as little interaction with the participants in order to fully benefit from the user / platform interaction. These interactions must be tailored in a way that reveals the product's issues and their origins.

In the same way, the creator-moderator has to be as impartial as possible during the product's presentation, in order not to influence the participants. Any attempt at justification, resolution or explanation offered to the user – about a functionality still being in development, for example – are to be strictly avoided. What has to be kept in mind throughout the test is that its goal is to highlight problems. The steps of understanding what these problems imply and how to solve them have to come after the test.

Finally, it is necessary for the participant to feel confident during the test in order to guarantee conclusive results. This implies formulating to them that it is normal to make mistakes and that there is no right or wrong answer. We are assessing the product's performance, not the participant's. If participants encounter difficulties during their journey, it will surely be coming from the product and not the user.

Generally speaking, the moderator will ask as few direct questions as possible. This will leave room for the participants to express themselves. The recommended questions are those which do not await a right or wrong answer. For example:

• What are you thinking about right now?
• You seem surprised / confused / frustrated. What's happening?
• Is there anything here that is different from what you expected?
• Do you believe that this piece of information should be provided?
• Is this close to what you were expecting?

It is important to treat each participant as individuals and to interact with them without taking previous exchanges into account.

In order to bring out as much information as possible from the participants, several techniques can be used:

Thinking aloud

Description

This method's objective is to bring out the issues that participants face through their commentary. This verbal protocol allows them to think out loud and to better understand their feelings during the test.

It is also important that the participants verbally convey their emotions for the purpose of later reviewing the test's video recordings.

Pros:

This technique allows you to record the participants' emotions and their journey without having to ask them questions.

Throughout the test, the participant provides clues as to what the sources of the issues are and what can be done to troubleshoot those issues.

The participants are usually more focused when describing their actions and experience out loud.

Cons

Certain participants might struggle to follow this method.

On the one hand, the “Thinking Aloud” process can slow down the test and add further strain on the participant's cognitive load. But on the other hand it greatly improves the quality of the obtained results.

How to improve this experience:

By experiencing it yourself: the creator/designer takes the time to think out loud when testing his or her own product.

The moderator can ensure that the participant feels heard by nodding or expressing approval at the user's remarks. This is a very effective way to encourage the participant.

By paying attention to and respecting the participant's pauses and silences: they generally signal that the user is focusing on overcoming an obstacle. This allows you to identify a friction point and to observe how this obstacle can be overcome.

The five “Why”

Description

Thanks to this easy-to-use method, it is possible to go all the way up to the source of each of the issues that the participant encountered. It allows you to identify the cause not through supposition but by having the user describe the issue he or she is facing. It is applied by asking the participant a set of questions, each reprising the problem raised in the previous question. By doing this, you can get to the heart of the problem and find its source:

Why can't you log on? I can't find the login button.

Why can't you find this button? I'm looking for it at the top of the page but there isn't any interaction available.

Why are you looking at the top of the page? This is where I usually find the login button or the access to my account.

By using a few quick questions you have been able to grasp what the issue was and find a suitable solution which could fit your product.

Pros

• It is easy to put in place.
• It allows you to go to the heart of the problem;
• It allows you to find solutions quickly;

Cons

• The participants can answer the same question in many different ways.
• It is necessary to take some time to condition the participant before asking the set of questions.
• It cannot be used too often as it could become frustrating for the participant.

Finally, it is essential to remember that the moderator should only help the participant as a last resort. This aspect is detrimental to the identification of usability issues on a platform. This is a tricky exercise because providing help to someone who is struggling is instinctive. The appropriate alternative is to encourage the participants to voice their impressions.

The times when the user faces difficulties will be the foundation which will allow you to formulate future improvement hypotheses. There is nothing more convincing to a sceptical designer or lead product manager than seeing a video recording of a user who is not able to carry out a task on their platform.

The Results of Usability Tests

Once all of the tests are run, we will present you with a detailed analysis. Using the data collected during the test sessions, we will create a general report which will highlight which participants had difficulty using your platform and which tasks the participants struggled with the most.
“See section – Comparative analysis”

You also have access to a detailed analysis for each of the tests that was carried out, so as to thoroughly understand where the participants succeeded and where they failed. “See section – Individual analysis”

Prioritizing Problems

After the major issues faced by the participants have been identified, you need to classify those issues and to find the balance between how much effort needs to be put into fixing them and how much you will benefit from it. It is also advised that you create a comparative list of the friction and satisfaction points that were identified during the tests.

According to NNGroup, you can prioritize your list using 5 degrees of importance for the issues that require your attention.

0 = False positive, the problem raised does not require any fixing.

1 = A graphic design problem, to solve if there is enough time to do so.

2 = Minor usage issue: to put in the 'necessary but non-urgent' resolution category. Those are issues that have to do with spelling, typography or spacing for example.

3 = Major usage issue: high-priority, important to solve. This can foster frustration during the use of your service or product.

4 = Usability disaster: it is imperative that this issue be fixed before the product's release. Top-priority. For example, the payment system is not working, or the user cannot log onto the platform, etc.

Making changes on the product/the procedure – an Agile methodology

It is time to take the necessary measures to improve the product. Ignoring or delaying this step will lead to usability issues. If the product is launched with these issues, there is a high chance that it will not be successful on the market.

Discuss the results with your team

The objective that is central to usability tests is to create or improve a product/service that your clients will use and appreciate.

It is important to include the different parties that were involved in the development of the product in the analysis of the results from the usability tests. Each member will have the opportunity to better grasp the issues encountered and to actively participate in fixing them.

Set goals for yourself and your team

Depending on the degree of importance assigned to the identified issues, it can be very useful to make projections of the impact that solving them can have for your product. The ROI methods will be useful for you prior to the UX research stage, but also when implementing your product's functionalities.

Additional testing for improved results

Once the necessary modifications and improvements have been made after having been identified with the usability test, it is particularly useful to repeat the research process, even with limited range, as it would allow you to check and confirm each iteration.

As of today, user experience methodologies do not allow for multiple tests on the same product. Because they are carried out manually, these methods are time-consuming, can become costly and do not fit the rapid production model that the market dictates.

Because it can be implemented instantaneously and is cost-effective, the solution we offer will integrate seamlessly in your Agile development and your swift user experience research process. If you are not already familiar with it, this introduction to UX research will allow you to better understand the aforementioned Agile methodology.

Lastly, if you do not know where to begin when it comes to using our software, please refer yourself to this part of the Odaptos user guide.

Useful resources:

234 Tips and Tricks for Recruiting the Right Users as Participants in Usability Studies - NN Group

How to Conduct Usability Studies - NNGroup

--

How to Plan, Design, and Conduct Effective Tests - Jeff Rubin, Dana Chisnell

A Practical Guide to Usability Testing - Joseph Dumas, Janice Redish

--

The Design of Everyday Things - Don Norman

Flow: The Psychology of Optimal Experience - Mihaly Csikszentmihaly