Labs/Ubiquity/Usability/Usability Testing/Fall 08 1.2 Tests
Objectives
Begin probing how to make Ubiquity accessible and useful in the context of including Ubiquity with the mainline Firefox distribution. As this is an early alpha I will be focusing on the core Ubiquity interface, while logging bugs on separate commands.
Methodology
See main article Methodology This largely is an exploratory, qualitative test exploring what users do when presented with Ubiquity for the first time. It is based on the interview format where the participants lead which tasks they perform next.
Analyzing Data
These tables are being pushed out before I have had a chance to have someone verify them and link them to the Trac DB. They are subject to changes, revisions, and mistakes. This is a wiki, so you can help out ; )
| Gives up (Discovery) | 5 | 3 | 4, 6, 9, 10, 11 | Discovery | Users who gave up on finding the hotkey combo. | 402, 440 | |
| Finds Hotkey | 3 | 0 | 3, 5, 8. | Discovery | Anywhere from 1-9 min. | ||
| Notices Hotkey Setting Field | 2 | 1 | 5, 8, | Learnability | #5 Didn't know what it was (and his two guesses were wrong) and 8 found it by accident, and changed it by accident. #7 accidentally changed the setting. | ||
| Changes hot key accidentally. | 2 | 3 | 7, 8 | ||||
| Problems with command structure | 1 | 2 | 10 | Learnability | I.e. the need for modifiers etc | ||
| Confused by Jargon | 5 | 1.5 | 6, 7, 11 | Learnability | Wiki, hotkey, mashup, | ||
| Confused by wiki | 1 | 2 | 6, 8 | Learnability | |||
| Tries Wikipedia for help | 2 | 1 | 9, 5 | Learnability | |||
| URL instead of email command | 3 | 2 | 11,4,9 | Learnability | Lead for stat analysis, synonym common email URL's w/ email command. | ||
| Tries Right Click | 1 | 1 | 9 | Discovery | Contextual menus have discovery issues, the one tester used it only because he read it in the tutorial. He never did open Ubiquity either.
(link to other studies on contextual menus) |
||
| Tries watching video | 6 | 6, 7, 8, 9, 10, 11 | Learnability, Value | ||||
| Video won't load | 3 | 2 | 6, 11, 10 | Learnability | External validity of this data is poor because all users were using the same wifi connection (ie not a random sample) | ||
| Other video problems | 1 | 1.5 | 7 | Video is buzzword laden | |||
| Video volume | 2 | 2 | 7, 10 | Sound |
Commands
commandname- number of unsuccessful attempts total
commandname+ number of successful testers total
If a command was executed subsequent successful attempts don't tell us anything. If someone screws up the total number of attempts and separate issues are counted. Think of it this way, there are an infinite number of ways to crash a plane but only one right way to land one.
Remember, these numbers (as is with all usability stats) are to bring a level of objectivity to what is inherently subjective observations.
| Email- | 4 | 9, 11 | Two seperate issues | |
| Email+ | 3 | 5, 6, 7 | ||
| Maps- | 3 | 8, 9, 10 | ||
| Maps+ | 4 | 7, 8, 10, 11 | ||
| Wiki- | 0 | |||
| Wiki+ | 4 | 6, 7, 11, 10 | ||
| Weather- | 1 | 10 | ||
| Weather+ | 3 | 5, 10, 11 | ||
| Define- | 1 | 5 | Sudo did not show up in Define.com. Fallback dictionaries (urban, wiktionary, Google's define:, etc) | 404 |
| Define+ | 3 | 5, 7, 10 | ||
| Translate- | 2 | 10 | ||
| Translate+ | 2 | 5, 8 | ||
| Help- | 1 | 10 | ||
| Map-insert- | 6 | 5, 6, 7, 8 | ||
| Map-insert+ | 2 | 5 | ||
| Yelp- | ||||
| Yelp+ | 1 | 8 | ||
| Google+ | 2 | 6, 7 |
Synonyms & new command suggestions
The email command received the most amount of trouble, especially when chained with inserting a map. Some hotkeys tried were just "control E" or typing the url into the command line i.e. email.yahoo.com. Tester 8 and 10 were prolific guessers.
- Lyrics 14:09
- Yelp insert- 21:14
- Find 18:44
- Locate 19:10
- Help add command 23:37
- New command 23:39
- Gadgets-Command list
- Find gadgets-Command list
- Update-Command list
- What's new-Command list
- Add command-Command list
- Stockprices
- -Stock Prices
- Ticker--Stock Prices
- Zipcode -Map
- Directions-Map
- Closest ie Closest japanese restaurant
- “map home” modifier of map command for location.
What really jumps out at me here is yelp insert and help add command. Should insert be a universal modifier? And help add command suggests that the help structure for commands would be centered around a command entry, like in Enso.
Demographics
The aggregate demographics of the participants are uninteresting. The reasoning for demographics was to provide insight to user behavior and thus the demographics and feedback of the tester are available on each session page.
The accuracy of the data is questionable. There is sampling bias, the previous survey our was modeled on was done at random over the web whereas these participants had someone next to them, creating social pressures. Test Pilot is probably the best value for studies that require statistically accurate sampling.
A couple testers commented that they worked full time using computers, so their answers should have been 40+ hours per week, but none marked it as such. Overall, was an even distribution from the "1-5 hours" all the way to "21-40 hours"
None of the users deemed themselves as power users, which could be part of the sudo-anonymity online forms provide. Half labeled themselves as "modest/beginner" one was "illiterate" and the rest considered themelves "experienced." I also feel that these terms are loaded and innapropriate for a survey, but they are what the previous study used.
Firefox usage history could have been disturbed by social pressures as well, although these matched well with the previous study. One was not a Firefox user, two had "Less than 3 months" experience, one had used Firefox for "1-2 years", and six had used firefox "Over 2 years".
The reliable statistics are OS and web service usage as there was social judgment about these questions. Seven used Windows, two were Linux users, and one OS X user. Everyone used email, Wikipedia, and a map/directions provider. Eight used Gmail and half had used a translation service.
A PDF of the demographics with pretty graphs is here, a zip of the spreadsheet in numerous formats is here.
Feedback
The feedback has internal consistency issues. While the majority of users said they would use the product the majority didn't think Ubiquity was very useful. Furthermore the facilitator's impression is most were frustrated Ubiquity. The participants have been emailed and we will see if they actually kept using Ubiquity.
Because of the sample size it's tough to draw solid conclusions from the data when the variables do not correlate.
As Dana Chisnell commented, "I wonder if what this says is that your participants thought it's an interesting idea, it could be developed more, but there isn't a lot of immediate value because it doesn't actually make their tasks easier."
How do you convey value to the users? Slashdot considers Ubiquity (or rather the "natural language interface") bloat. Granted this IS /. but tech fanboys are the base that installed FF on millions of machines and taught others to use it. It's a perception issue, it needs to be fixed.
Is there something that the help is lacking?
Some participants chose not to leave feedback, 3 and 4 did not have those questions on the test.
- 5 Break into multiple pages, not just one big scrolling adventure.
- 6 Better images, less words. Working movie.
- 7 No phone numbers to call, video is a bit dry, web page could be spruced up a bit visually
- 8 Easier searches, maybe a help that brings up lists of help options like in ubiquity - or have help directly in the ubiquity window so you don't have to stop browsing to find out how to do something.
- 10 If it needs help, it;s not psychically reading my mind enough. I've never used help for any other software.
Tester 10 has a good point, even if it is a bit juvenile. It is something that every usability professional told me when I asked for advice: if the user needs the help section (however good that section is) you have already failed.
Suggestions, thoughts, Kudos
- 5 Would like some sort of window persistence / docking functionality. Inserting non-native content onto any page you like (map insert) might be a bit presumptuous.
-The inserting of content comment was concerning copyright issues.
- 6 It might be hard for people to learn how to use the tool without extra help. It is confusing for people who don't know how to program or use computer at a high level
- 7 a very interesting addition to mozilla browser. It seems to make browsing/searching/emailing much more efficient.
- 8 Make commands easier to edit, or have ubiquity understand what you're trying to say. Editing and adding new commands within the ubiquity window.
- 10 no one is ever gonna get using ctrl + space to pull it up. Please make it part of the awesome bar or a bar that is on all the time.
Insights
Everyone takes away something different from each session and those understanding really contribute a lot to our overall understanding. Please contribute to this section heavily.
Discovery
Skipping the hot-key information transcended across user competency levels, of the two who didn't discover it by accident one rated himself computer "illiterate" and the other "experienced." Their mutual key to success was reading the documentation carefully.
Discovery will be enhanced by placing Ubiquity into the URL bar.
Marketing and the possible use of contextual notifications on screen would help to expose users to Ubiquity and it's interface outside of the browser experience.
Demonstrating Value (selling)
Learnability
The tutorial saw a lot of skipping around (Trac #404), which tutorials are not supposed to accommodate. As Tester 8 put it, "[reading from the intro page]'Read the ubiquity tutorial' -that sounds boring." Moving towards an Enso-like approach to help (where each command page has instructions) would be beneficial. Enso's help system has undergone usability testing.
The video was also subject to skipping around too. However, scrubbing through long video is too clumsy to be used as a reference. An intro video and shorter, command oriented videos might prove beneficial.
It's hard to gauge how long people stayed tuned in to the screencast. They would often experiment and then come back. It would be good to get some analysis with a little JS to track how long our current user-base stay on the page/tab after clicking play. Stay tuned for some [www.indolering.com blog posts] on thoughts concerning video.
Packaging Ubiquity for the Mainstream
What does this study teach us about packaging Ub for the mainstream?
UI Triangulation
What it is. Recommendations for integrating it into dev cycle.
Continual Improvement
TPS continual development cycle/ turning everything into a scientific experiment, implementation in software, measuring success of changes.
Streaming UI
The potential for it.
Sessions
Tests 1 & 2 were dummy tests, 3 & 4 were lost due to encoding errors.