Five users will find 85% of the usability problems – WRONG

According to Rolf Molich of Dialog Design in Denmark.   He has been conducting Comparative Usability tests with usability experts over the last 10 years and has found that different experts find different usability problems.  Also, that contrary to Neilsen’s much cited quote of ” you only need to test with five users’  he says this is nowhere near enough to undercover the majority of usability issues.

Nielsen usability testing with Five users

Rolf says however that 5 users is useful number to gain

feedback for iterative usability testing cycles. 

He told us this and more at a conference I recently attended in London UCD2012.  Some other interesting points from his talk;

  • Research has shown that Expert Reviews found as many usability problems as usability testing** caveat here – not everyone is an expert
  • Remember to include positive findings in a research report – so that features that users like are not pulled by design team
  • And as he says; “when you have a hammer in your hand – everything looks like a nail” -Remember it’s critical issues you are testing for
  • Lab testing vs non lab testing  – Rolf says one of most useful aspects of testing is that stakeholders can see the user engaging with the product; it doesn’t matter if in lab or not in lab, as-long as stakeholder can see what is happening
  • Beware users opinions – they are that – just opinions.. better focus on facts – what the user does

A very interesting talk, though as a usability researcher of 5 years or so, I still think a smaller number of test participants 5-8 can be very useful in uncovering the top usability issues, if not 85% of them.

But of course agree that test little, test often is the way to go with regular iterative testing cycles.


One thought on “Five users will find 85% of the usability problems – WRONG

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s