“When you have a usability hammer in your hand – everything looks like a nail” – Rolf Molich
How many participants in a Usability Test ?
Testing with one user is 100% better than testing with none – Steven Krug.
We have all seen Nielsen’s graph which says 5 participants is the optimum number to uncover 85% of usability problems
If this sufficient to measure outputs? Jeff Sauro has a great blog called Measuring usability – which gives a huge amount of info and advice on how to measure usability. He tells us if we want to ensure a task-completion rate is at least 70%, we should plan on testing at least 8 users.
And it depends on what method you are using. For eye-tracking and particularly if you want to use statistics on results – Neilsen recommends 30 participants.
Personally, if it is one day of general testing and you are not comparing specific designs – I think 6-8 users with a test length of 45 mins, with – 5-10 in between (to jot down main points), a good number. Anything over this number and you start hearing the same issues over and over..
I was recently at a UCD conference in London where Rolf Molich gave a talk on “debunking myths about usability” and Neilsen’s 5 users graph was one of the myths on his list. Molich maintains you need more, a lot more users to undercover 85% of usability problems . However, what he was recommending was smaller, more frequent tests at different stages of product development and for this he mains 3 or 4 users are sufficient – for iterative continuous testing.
Test little and test often.
What to test in a Usability test:
This was always the biggest question for me. I found, especially working with an agency, that clients would come in and say – something is not working on our site and we want a usability test of our site –
It might not even be a usability test they need (may be more discovery piece). And to remind ourselves what usability is (according to Steve Krug in Don’t Make Me Think)- “Usability means making sure something works well and that a person of average ability or experience can use it for its intended purpose without getting hopelessly frustrated”.
If it was a usability test needed, the next step was to try and narrow down what exactly it is they should test – particular function, area or concept.
This is where analytics can help us focus our efforts. It can help us identify problem areas in the site that we may need to look at further.
Another approach when considering what tasks to test is to consider – What are the top three most important things people need to do when using this site (Steven Krug)
It is worth spending time on this part of the project to ensure research outputs meet business needs and expectations.
How do we measure Usability?
Jeff Sauro talks about task completion rate percentage. When testing, the version I have used of this is not a percentage perse but rather a ‘ranking’ of task in terms of how successfully they completed it.
1 = without hesitation, 2= hesitation/alternative route 3= intervention etc. And I have found this useful.
Of course the SUS satisfaction questionnaire has been aournd for a long time and while the language may be a bit dated, analysis by people like Tom Tullis shows that this questionnaire is still probably the most consistent after- test questionnaire. The problem is of course, that especially at the end of the test, participants rate tasks highly even though we have seen them struggle. (Jeff Sauro – Around 14% of users who fail a task still rate it as super easy to do)
The way we get around this is to rate satisfaction, ease of use etc after each task which I think improves the feedback. And one measure we have been trying out recently to capture this is Jeff Sauro’s Single Measurement Score (Below).
And of course Time on Task is an industry standard. Using Noldus I have found this easy enough to collect during testing. Anytime I have tried on Morae, it upset the software. Often, we have found this measure unnecessary to collect, especially if it is already obvious the participant struggled with the task. It can be useful though, when looking at the ‘Findability’ of different elements or when considering reactions to different designs.
What about emotion?
We have been trying a couple different methods recently to look at emotion during testing. Such as asking participants to tick different faces that best describe their feeling to asking them to look at a stick man and describe what is he thinking (about the product). We have had mixed results and I think the most important aspect is framing of question – we are working on it!
Interviewing in Usability Testing
In an interview – participant should talk 95% time and the interviewer 5% of the time – John Waterworth from Foolproof
The Interview is an important part of the usability testing– the value of usability testing is that you get data from several sources e.g observation, eye-tracking and interview – which helps a truer picture of the experience emerges.
Some questions I have found useful during interview
- What are you thinking?
- What is going on here?
- What are you looking at
- What is going on here
- What are you looking for?
- If you were at home, what would you do now?
- Describe what it is and why use it friend (new product)
It is not advisable to ask ‘if you had a magic wand what would you’… participants may just make information up.
Or to ask participants how does X make you feel – it may not make them feel anything at all! Better to repeat what they say when talking about emotions e.g participant A – this is really boring – Interviewer – Boring?
If the participant is a mumbler (and there are many!) it is important to repeat or clarify what they are saying – both for yourself and any observers in the back room. I have also found it useful to ask them to use the mouse when explaining different aspects of the screen – useful again for observers and also when going over screen recordings later.
Final Thoughts on Usability and UX research
Usability lab testing is expensive. A couple of areas that I would like to further explore:
I am keen to experiment with (evermore popular) online usability testing products. Not to replace traditional lab testing but to see how it works along side it..
And i would like to work more with other research methods (both online and off) to give further depth to usability findings. And find out which methods/tools work and which ones are less useful
And to consider in-house lab testing – how does that work- does it impact validity?
To be continued…
Eye tracking Web Usability – Jacob Nielsen
Don’t make me think – Steve Krug
Rocket Surgery Made Easy – Steve Krug
Beyond the Usability Lab – Tom Tullis