The pros and cons of online usability testing

Pros

 

The pros are pretty easy and mostly centre around cost and ease of implementing.

Cost

You don’t need fancy usability labs with 1 way mirrors, facilitators, observers, cans of soft drinks, snacks for the participants, a finder’s fee for each participant and of course a fee or gift for the participants. 

www.usertesting.com charge $39 (USD) per user, a fraction of even the finder’s fee for traditional user testing. Undertaking user testing at full lab would normally cost you at least $1000 (AUD) per user.

Implementation

The implementation is quick and easy by following a simple setup wizard on the relevant site.

Amount of testing you can afford

An indirect second benefit of the reduced cost is that you can afford to do much more testing, testing of your existing site, your proposed solution (at multiple stages) and your final product, for a fraction of the cost of just 1 round of traditional user testing.

Comfort factor

Users are participating in the testing in a more natural, real world environment, where they feel more comfortable and are more likely to be themselves (no matter how many time you tell a user you’re testing the software and not them, they still feel under pressure to get the answer ‘right’ and you can’t blame them really, when they are being videoed and watched through a one way mirror ;-). 

Cons

 

Testing junkies

Testing junkies are there just for the cash and take a number of tests (one of the criteria for traditional user testing is that you haven’t participated in user testing before, as it devalues or alters your input).

An accurate audience

Getting the correct audience type for your testing is vital. Traditional user testing companies usually outsource the finding of users to specialist firms who (for a fee) will find you test participants of the correct sex, age, demographic, psychographic and even to the level of whether or not they have used your products and services before. 

Security

You’re less in control of the physical environment and unable to stop users taking screen grabs or otherwise sharing what they are seeing. If you are testing a new to market product, how can you stop your competitors from hearing about it or even worse testing it?

Body language

Testers quite often come from a psychology background and can learn a lot from interpreting a participant’s body language. This is lost altogether with online testing.

Remoteness

It’s hard to hand somebody a physical hand out or flyer and ask them to go online and do what they would normally do in the real world if they’re half way round the world in a different country.