Bugzilla:CMU HCI Research 2008
In the Fall of 2008, a group of students getting their masters degree in Human Computer Interaction (HCI) from Carnegie Mellon University in Pittsburgh volunteered to do some user research on Bugzilla, for credit in one of their classes. This group consisted of 5 students:
- Tom Bolster
- Fred Pfisterer
- Emily Vincent
- Vedant Mehta
- Josh Zúñiga
These students conducted the following HCI usability research methods:
- Contextual Inquirty
- Keystroke Level Modeling (KLM)
- Cognitive Walkthrough
- Heuristic Evaluation
- Think Alouds
All of this research was done on the 3.0 release of Bugzilla.
Contextual Inquiry
Contextual Inquiry (CI) is an observational method which focuses on users performing work within their environment. The observer asked the users questions, as they were performing the task of interest, to collect rich qualitative data. The clear benefit is seeing the minutiae and understanding why those details matter in the context of the user’s work domain.
The student group conducted five CIs during their research. The first CI was with a resident assistant whose responsibility was to support the dorm residents during their stay for the school year. As a part of these responsibilities, this resident assistant reported maintenance problems using a web based system. Our group wanted to explore the concept of trying to get help with something that an individual could not fix themselves. This CI aligned with our focus; enabling sustained involvement of new users in defect submission.
The second CI was conducted with a documentation specialist/tester reporting problems end users experienced with the product they develop. This user was the connection between the development group and the end users of the product. The user used a competitive defect tracking software to Bugzilla.
The student group conducted the final three CIs with different individuals in the same company, but with widely varied roles. This company uses Bugzilla as their primary defect tracking tool to aid in their software development. The first user role we observed was a developer who had not used Bugzilla before. This user would be considered a skilled developer. The second user was a development lead for the group and would be considered an expert user of Bugzilla. This user was responsible for planning and guiding the development efforts of the product. The final user observed was an application tester who was responsible for application level testing and QA of the product. This user would be considered a novice user of Bugzilla.
During the modeling stage the group developed models which illustrated how users went about their work in context. Through the models we showed (1) how communication flows to and from users, (2) the sequence of actions to complete a task and the underlying intent, (3) the cultural influences on the user from people, groups, or objects, (4) the artifacts and how those were used, and (5) the physical space where the user performs their work. Each of the models revealed important findings around the user’s work. Contextual Inquiry was by and large the compass the team used for guidance as it allowed them to abandon their assumptions of the user and see how people actually behave in context.
By nature, CI has high preparation costs associated with it as is highly qualitative and subjective. In this instance five users were observed, but ideally a group would collect upwards of 15-20 CIs of many users involved in the work being observed. Forming a focus to guide questions, pre-interview practice, conducting the interview, modeling, post-interview follow up, and finally consolidation and interpretation sum to a cost far greater than each of the other methods we used.
In comparison to other methods, CI is an input to the other methods. Inherently, CI is used to understand the users and observe what they do in context. CI is a guide for design and does not require that a system even exist in order to perform. In contrast, the other methods are used to evaluate a system as it exists in low or high fidelity.
Keystroke Level Modeling (KLM)
Working on this section
Cognitive Walkthrough
Working on this section
Heuristic Evaluation
Working on this section
Think Alouds
Working on this section
Conclusions
Working on this section
Bugs Filed To Respond to Research
Working on this section