UXBristol Special Workshop 2: Measuring Usability with Rolf Molich

Qualitative usability testing is by far the most widely used usability method. However, usability practitioners are discovering that they need to accommodate managers who are no longer satisfied with qualitative data and insist on measurements because "You can't manage what you don't measure."

The current literature on usability measurements aimed at usability practitioners is very limited. This course offers practical information about best practice in usability measurements with real users. The course gives rare insights into how experienced usability professionals do measurements. The course draws upon findings from the recent CUE-8 study where 15 teams including some renowned experts independently tested the same web site under realistic conditions.

What you'll learn

• Good arguments for measuring usability
• What to measure and how to measure it (and when it's not worth measuring)
• How many participants you should test and how the number affects uncertainty
• What to look for in measurement tools, particularly automated measurement tools
• How to communicate measurement results
• Typical costs of usability measurement

Course Outline:

• Introduction
  • The CUE-studies, in particular CUE-8 about measurements
  • Exercise 1: Survey - Your views on usability measurement
  • Resources: Articles, books, websites, forums, services
• Why you should measure
  • Good arguments for measuring usability: You can't manage what you don't measure
  • Usability metrics vs. site analytics: what each type of numbers is good for
• What you should measure
  • Time on task
  • Success/failure rate
  • Disaster rate
  • Satisfaction
  • Other metrics
• How you should measure
  • Overview: How to conduct a usability measurement
  • Differences between qualitative and quantitative usability evaluation
  • Measuring time on task
  • Measuring success/failure rate and disaster rate
  • Measuring satisfaction - Questionnaires
  • Local versus remote testing
  • Attended versus unattended testing
  • How to handle failed tasks
  • Key pitfalls in measuring usability, and how to avoid them
  • Quality assurance
  • How to create great measurement tasks
  • Exercise 2: Test tasks for measuring usability
• Analysis
  • Composition and size of participant sample - Number of participants
  • Variability from samples - Uncertainty
  • Exercise 3: Analyze videos from real measurement sessions
• Communication of results
  • Quantitative results
  • Qualitative results
  • Useful and usable recommendations for improvement
  • How to measure and prove overall progress
• Automated measurements
  • Automated measurement tools - A consumer's guide
  • Sources of error
  • Extreme versus incorrect results
  • Killing the ugly ducklings: Data contamination and how to prevent it
• Cost and time of a measurement study
  • Usability measurement versus traditional usability evaluation

Format

This full-day course includes lectures, videos from usability measurements, discussions and some exercises.

About Rolf Molich

Rolf owns and manages DialogDesign, a small Danish usability consultancy that he founded in 1993.

Rolf conceived and coordinated the Comparative Usability Evaluation studies CUE-1 through CUE-9, in which more than 100 professional usability teams tested or reviewed the same applications.

Rolf was a principal investigator in the Nielsen Norman Group's large-scale usability test of 20 US e-commerce websites, involving more than 60 users.

He has worked with usability since 1984 and wrote the best-selling Danish bookUser Friendly Computer Systems of which roughly 30,000 copies have been sold. The book is now available in English, with the title Usable Web Design. Rolf is also the co-inventor of the heuristic evaluation method (with Jakob Nielsen).