Support/Goals/UX Testing/2010-12
Since I did the last round of user testing in October, we launched the new KB with a redesigned start page, search summaries (and keywords that work), and a bunch of search fixes (bugs 586034, 596623, 596624, 596973, 601655, 586034). The good news is that the front page search is much improved. If you search the problematic phrases from the last test you'll see that you get much better results now:
- Why did adobe flash crash (previously this returned zero results)
- What to do when flashplayer crash (previously this returned zero results)
- flashplayer crash (previously this returned only forum threads)
- flash player crash (previously this didn't find the article "The Adobe Flash plugin has crashed")
- can't find downloaded files (previously didn't return any relevant results)
For this test I repeated the test from October. This time the tester was able to easily find the correct article each time with the target articles listed in the first few results.
In addition to testing the front page search, I tested to see how people found SUMO by starting them on the Firefox start page and telling them to find an answer for their issue. One tester went directly to SUMO by choosing Firefox Help from the Help menu and the other searched Google. Though both testers found SUMO relatively quickly it still seemed to happen a bit by accident in each case. The first tester was at a loss for what to do and tried looking in the Help menu in desperation. The second tester's Google search brought up a relevant site that wasn't SUMO. It only had instructions for Windows and since he was using a Mac he couldn't make it work. So he went back to the search results and chose the correct SUMO article which was the second result. I think that working to make SUMO more visible still needs to be a top priority.
I also saw a few other issues:
- Generating a good search query is often more difficult than matching your problem to a list of articles. The tester who was allowed to do more than just search from the front page found that almost every issue was listed in some way on the front page and preferred that to searching. And when she did search she even ran into difficulty using the search form itself. video (1:30)
- I've been bothered by the plugin crash UI and our documentation around it for a while. One tester, mistakenly thought that the crash report would be sent to her. She also could not make sense of the documentation and said that if this had happened at home she'd "just shut the computer off or re-install Adobe Flash."
