How many users does it take to…?
As UX’ers undertaking usability projects we often get asked how many participants do we need for usability tests to ensure that results are reliable and actionable.
I have blogged about this before but it is a question that keeps popping up so I wanted to look at it again.
First off, what do the experts say? Nielsen.
This says 5 Users will uncover most important usability issues and after that testing falls foul of the law of diminishing returns; each additional user will give you less and less insight.
Rolf Molich who worked with Nielsen on 10 Usability Heuristics For User Interface Design (published 1995 but still relevant!) disagrees with this statement, arguing that it takes far more users than 5 to find 85% of all problems.
Molich has undertaken many years of research in his CUE Studies looking at how usability professionals evaluate technology. However, Molich does continue on to say that if we are trying to sell on usability to an organisation or if we are planning iterative continuous testing 3-4 users may be enough for each round.
Neilsen agrees that breaking a test of 15 users into 3 different tests is better than one big usability test when when testing iterative designs.
Jeff Gothelf in his book on Lean UX sites the example of Meetup research team testing with 3 participants once a week on a Thursday concluding that every round of testing ‘brings us closer to the truth’. Regular testing is an essential part of agile development.
A small number of participants is fine (3-5) BUT
Test early and Test often (and then Test again)
In his book on usability testing Steven Krug says ~ Testing with 1 user is 100% better than testing with none~
This chimes with Nielsen curve, above with shows that zero users give zero insights
I am not 100% sure i agree with this. What if that user is, for some reason different to other users?
The rapid iterative testing and evaluation (RITE) model developed by Microsoft does also allow for changes to a design after one test and one user BUT that system has checks in place to counteract any outliers. RITE Methodology is another useful tool for Agile.
My Top 5 Tips on usability testing and participants
- Dont get hung on the numbers but instead spend time, energy and $$ to get the right participants who are representative of target group
- Screen those participants (or make sure Recruitment agency does) to ensure they are articulate and can voice their ideas about what they are doing/thinking.
- Frame testing/research aims as hypothesis; something that you are testing to prove/disprove.
- Combine usability testing with analytics; both prior to testing to inform test plan and highlight problem areas and after testing to test hypothesis/findings
- And of course ~ Test early and test often and then retest
There is no magic number for me but if it is a once off usability test (with no planned follow up tests) I have found recruiting 6 participants and 1 standby to work well. With 6 people we generally tend to see the same top line issues emerging and also importantly this number allows us some time in between each test to discuss results with clients and observers and/ or to tweak test plan for the next participant.
> More Resources <
10 Usability Lessons from Steven Krug’s Dont Make me Think