Government Leader home > August 2005 issue
August 2005; Vol. 1 No. 3
Identifying high-impact user problems with usability testing
Then came the hard part. Hylton and his team developed a cause-and-effect diagram. With lines and arrows and tiny type, they mapped out how people, materials, equipment or policy contributed to what Hylton determined amount to “defect calls.” Many of them were attributed to customers—and call center employees—not having the right information at the right time: a particular form, clear instructions or a current phone number, for example.
Next, Hylton’s group developed a series of scripts and “gadgets” to make it easier for customers to navigate toward the answers they needed, make online retirement estimates on their own or get answers to common questions.
But the underlying question for Hylton was, how usable and effective would these approaches be? And how would they know?
That’s when Hylton tapped into the Usability Lab, a resource center run by DFAS Shared Services Center. The lab is a working repository of usability research and application tools, based on work available at a variety of institutions and Web sites. Among Hylton’s favorites: www.usability.gov, maintained by the Health and Human Services Department; www.measuringusability.com, and www.useit.com, a site maintained by Dr. Jakob Nielsen, who specializes in Web-design usability. It was Nielsen’s work that helped Hylton realize that nine out of 10 usability problems can be identified using just six test-users.
Usability involves a combination of factors, said Hylton: ease of learning; efficiency of use; memorability; error frequency and severity; and accessibility and subjective satisfaction. To test their ideas, Hylton commandeered two connecting rooms at DFAS’ Indianapolis offices and set up a camcorder, PC cam, TV monitor and three computers loaded with screen-recording software. That allowed Hylton to record task completion times; the number of clicks users needed and pages they viewed; success and failure rates; and post-task and post-test satisfaction scores.
After just a few design iterations, Hylton saw big improvements: Customer search times dropped 46 percent, failure rates dropped 58 percent and the number of users who gave up dropped 77 percent. Customer satisfaction scores, meanwhile, increased 104 percent. More importantly, Hylton said, after the new approaches were implemented, monthly calls handled by call center representatives dropped 25 percent, despite a rising number of calls. The calls representatives did get were more complex and took longer to answer, Hylton noted. But representatives now had more time to address them. And that resulted in increased customer satisfaction and higher productivity.
There were hard savings, too. By being able to put benefit employees on other tasks, DFAS estimated it reduced its costs to support the three-person call center by $140,000 in 2004 over 2003, a drop in the ocean for an agency the size of DFAS.
But that misses the point. The rapid turnaround in performance at SSC provided one in a growing number of cases inside DFAS about the broader potential performance improvements made possible by applying Six Sigma, lean manufacturing and usability tools to day-to-day operations.
That’s already being reflected in the number of government managers requesting proposals for Six Sigma training.
“We saw three or four [requests for proposals] a quarter in the last year” coming into Motorola University, said McCarty, “compared to one a year in previous years.” And the requests were coming in “from across the federal government, not just the Navy and the Air Force.” These agencies, including the General Services Administration, “are not talking about training black belts, but about organizational transformation,” McCarty said. He added that what DFAS is doing represents a model many businesses could learn from.
Hylton laughed about the apparent irony: “Government often looks to the private sector for innovations.” Who would have imagined “the government’s logo should be up there alongside Motorola’s as an innovator in Six Sigma?” n
Path to Improving Usability
A FEW GOOD USERS: The majority of usability problems can be identified by fewer test-users than is commonly thought. Nine out of 10 usability problems can usually be identified using just six test-users.
Measuring usability
Usability can be described as a measure of the quality of a user’s experience when interacting with a product or a system.
Common Criteria
Usability is typically reflected by:
Ease of learning
Efficiency of Use
Memorability
Error frequency and severity
Accessibility
Subjective satisfaction
Test Metrics (for Web pages)
Task search and completion times
Task success rate (succeeded easily, with difficulty, or failed)
Failure rate (number of times users gave up divided by total number of tasks)
Number of clicks
Number of pages viewed
Post-task satisfaction
Post-test satisfaction
Source: Dr. Jakob Nielsen, Nielsen Norman Group (www.nngroup.com; www.useit.com)
Page 2 of 2
<< Previous |
1 | 2 |
|
![Government Leader[purity]](/images/govleader_smlogo.gif "Government Leader [purity]"){kind=small}
