Home

Scaling Up Usability Testing In Drupal

6 May 2008 - 12:06am — Bevan
Still not sure why usability testing is important? See what a linux hobbyist found in an informal usability test on Ubuntu.

Usability testing (hereafter "UT") is getting a lot of attention in Drupal -- and rightly so IMHO. If we aim to have "100%" test coverage on Drupal's code and functionality for Drupal 7, it stands to reason we also need test "coverage" on Drupal's usability. If not, it's far too easy for bad interfaces to be developed that pass all the functional and unit tests, but fail miserably in the real world!

Jimmy Berry (aka boombatower) of GHOP fame plans to take the initial (but giant) steps in this direction in the Google Summer of code 2008.

UT Jargon:

  • The software is what is being tested (e.g. Drupal).
  • The evaluator is the user using that software (e.g. a random person).
  • The Usability Testing Engineer (hereafter "UTE") is the software developer(s) or other person(s) who design and implement the usability test for the software and the tasks and scenario for the evaluator (e.g. a Drupal developer, a usability professional).

Jimmy's project is to develop a Usability Testing Suite (hereafter "UTS") to quickly create "unit tests" for Drupal's usability. The UTS will implement a framework for gathering data from the evaluator's UX and interactions with Drupal and the web browser.

The UTS will be limited to relatively informal tests and limited in it's range of data collection, but it will be able to provide wider and more extensive coverage than any other type of UT.

DROP student drpuser made 2 great videos of his informal UT on Drupal 6' installation UI that demonstrates the value of informal UT.

The UTS will make this type of informal UT easy to deploy and standardize on a large scale, and allow for split testing, to test UI ideas and prototypes, and even "regression testing", to ensure usability is not degrading.

The initial version of the UTS will only gather basic data like a Drupal log (e.g. FAPI events, current uri, timestamps etc.), mouse or click tracking (with the Click Heatmap module, created by Jimmy as a GHOP task) and live feedback by text (like one-way IM). Future versions will be able to record, collect and analyse more data through UTS' extensible API. Such extensions to UTS could capture screen recordings, static screenshots, camera and microphone recording and more.

Further, the UTS will be usable for any project built on Drupal. This means that the UTS could be configured to test out any Drupal website or webapp to find out why member-conversion rates are low, what the users' pain points are or why something is difficult to use. And all without the high costs of a UT lab!

And still further, since the test occurs remotely and not in real-time, the evaluators (users) can be volunteers or real site users, and they can do the evaluating from their regular computer and environment.

This is not to say informal testing and the UTS are a replacement for formal UT and UT labs. Formal UT and UT labs have other advantages.

This last point is important. An evaluator in a usability lab can feel uncomfortable with the physical environment, unaccustomed to the hardware or software they have been provided to do the test or even pressured to perform -- as if the evaluator him/herself were being examined (even though steps are taken to avoid this). This can easily skew results and data in formal UT. Informal UT, and especially remote and non-real-time UT can reduce these risks to nearly negligible. This gives clearer and cleaner results.

I look forward to working with Jimmy as his mentor on the project. Bring it on SoC 2008!

Cross-posted from my blog at CivicActions: http://www.civicactions.com/blog/scaling_up_usability_testing_in_drupal