Hi. Glad you made it back. This article is the third in a series for setting up and running usability testing sessions to improve your in-development software. In the previous finely crafted articles I covered why you’re running the session and how to find guinea pigs — uh, test participants. Today we cover how to be ready for those participants when they show up.
First, you need something for them to work with. Generally, you should use the most recent version of the software, even though it’s likely only partially complete. Note that while usability tests can be done with versions as late as beta releases or as early as paper-prototypes, by testing early in the development cycle you can implement changes when they’re cheaper. (It’s very inexpensive to erase and redraw on paper rather than re-coding an existing GUI.) Further, you should test at every major stage of development — you can choose what a major stage is — so that you can a) continually check your progress, and b) implement your changes in a timely manner, when it’s cheapest to do so.
Once you have the software, you need to give your users some tasks to complete while using it. Since you’re looking for problems that will occur with normal product use, you need to choose typical tasks. A typical task for testing a word processor might be to write a short document, alter the formatting in some way and then spell check. A subsequent task could be to modify the page settings, preview the finished document, save and print.
Depending on the completeness of the software, your tasks may be more granular (open an existing file for less developed software), or more general (import a database and complete a mail merge for more developed software). Selecting appropriate tasks could kill an entire article, but here are some tips: first, don’t make the user waste time thinking about trivial things. For example, don’t simply say, “Write a letter,” as this requires the user to think of something to say, and the actual text doesn’t matter — so just provide some sample text. Same for coding and so forth.
Second, ensure that design choices you are particularly concerned about are specifically investigated through the chosen tasks. For example, if you think the debugger’s hard to use, have the user debug something. Don’t forget about the installation process. As the last coded component it often gets rushed, yet it forms the user’s first impression of the product.
Support documentation is another area that is often neglected at review time.
Overall, try and make the tasks ones that the user would actually typically perform, rather than just picking tasks that shows your software in a good light. That’s marketing’s job, not development’s. Also, word the tasks carefully. If you want the user to add a row to a table, saying, “Insert a row” will clue the user to look for an “Insert” menu, or at least search the docs for the keyword “Insert.” Try and use non-informative language.
Finally, let’s look at the equipment you’ll require. All you really need is a computer that will amply run the software, and maybe chairs for your users (unless you’re testing their ability to simultaneously type and stand). You should also consider having some way to permanently capture the session, to allow you to later review it more thoroughly and at your leisure, and to point out specific concerns to co-workers who weren’t in attendance. Possible mechanisms for recording the session include having a camcorder pointed at the screen (probably the simplest), recording the monitor signal directly to videotape with voice recording of the test user (this will require some special hardware), or recording the screen action directly as some sort of AVI file (this will require substantial memory).
Depending on how many co-workers want to observe the session, you may want to setup a separate observation room by splitting the monitor signal from your test machine and setting up a second monitor in the room next door. You should really also have a microphone by the test computer hooked up to speakers in the observation area, so that the observers can hear your users cursing and screaming.
You may also want to have a spare monitor on hand, in case your test subject punches one.
Next issue we’ll look at how to run the actual session.
Blau is a software designer working in the User-Centered Design area at the IBM Toronto Lab. He can be reached at ablau@ca.ibm.com.
