QA/QAC
- QAC Redesign Document
- First draft created 06-25-09 by Clint, Heather, Aaron - This is a work in progress
Contents
Overview
The current Quality Assurance Companion extension is pretty broken in terms of functionality and usability. We have had a "redesign" effort under way for many months now with only slight progress to show for it. It is time to change this and make using the QAC into a really enjoyable, useful experience for our community testers.
On Monday of this week (06-21-09), Heather, Aaron and Clint hashed through some use cases and took a hard look at the QAC, tab by tab. What follows is our lists of improvements, and it will be easier to follow along if you start up the QAC right now so you can look at the current interface.
Primary Design Considerations
The primary audience for the QAC is a new user to Mozilla QA. In the light of that the more 'user friendly' we can make these interfaces the better. Additionally, the more streamlined all these interfaces can be, the better experience that person will have in terms of not being confused and being able to easily and quickly contribute their time to working on testing.
We still believe that the original purpose of the QAC - to help people run litmus without impacting the browser (or the application under test) is a valid one. So, we are still going to be our own free standing window, and we will continue to constrain ourselves to the smallest possible screen real estate so that the application under test can take up more real estate. We hope to allow the user the ability to run the QAC and the application under test side by side on their screen.
The Startup Experience
The multiple panels that first launch when a user starts up QAC for the first time are a completely broken experience. The only useful thing those panels do is alert the user to some security flaws in the product (which we will simply fix) and obtain a login/password for the Litmus web application. The operating system/platform selection can be reasonably guessed and does not need to be displayed. So, in light of these thoughts our propsal for the "first run" is to bring up the QAC, but overlay a little dialog over the "Q" tab to allow people to create a login. It would look something like this:
====================================== | | | | Q | Litmus | Bugs | | | | | ------------------------------X- | | | Enter/create your login | | | | | | | | Login |________________| | | | | Password |_____________| | | | | | | | | [Submit] | | | |_______________________________| | |_____________________________________|For comparison, the old dialog window:
Now, this is a little simplistic but you get the idea. What we really want this UI to do is two things: create an account for you and log you in to an existing account. If it is creating an account for a new user we want to automatically create a login for the top three sites that a new user will need:
- Litmus.mozilla.org
- Quality.mozilla.org (QMO)
- bugzilla.mozilla.org
In order to work for existing users who already have accounts for these items, then the "log in" section will probably default to logging you in to litmus and not the other accounts. Obviously we'll need some messaging around this so that people will know that they are now the proud owner of three happy QA related accounts. There is a whole host of complexity here, especially with existing users who may have different addresses for their user names on each of these three services.
Overall UI
You might have noticed in my little mock-up above that the "Chat", "Settings" and "Help" tab are removed. We decided that this material doesn't constitute enough value to merit screen real-estate on their own.
The Chat and Help items will be moved into the Q tab, the Settings tab will be moved into the standard application preference pane where it will take up its own tab there. Those preferences will be accessible from both the "Preferences" pane window as well as the "Preferences" button off the Add-Ons dialog.
The Q Tab
The Q tab is the news and events area of the tab and serves as a little mini-feed reader for the content hosted in the quality.mozilla.org (QMO) calendar and blog. We will add in the "Chat" functionality as a link on this page and the help button as well. It might look something like this:
====================================== | | | | 'Q' | Litmus | Bugs | | | | | - Events --------------------- | | | New Testday Oct 22! | | | | Firefox 3.5 Released! | | | |_____________________________| | | | | - News ----------------------- | | | Blah blah blah blah | | | | lorem ipsum dolor... | | | |_____________________________| | | | | [Chat with Mozilla QA!] | | | |_____________________________________| |_________________________________[?]_|For comparison, the old dialog window:
So, this is pretty similar to what we currently have. The "Chat button" will start up either the ChatZilla instance we currently ship with the product or will create a mibbit window and log you into #qa on irc.mozilla.org. There is also thought that the "Chat" button might be a multi-faceted control so that the user can choose whether to use IRC or simply send an email form to a known address if they are too timid to jump on IRC.
The [?] button will be available at the bottom of every tab and will cause a link to be opened in your browser to a help page hosted on QMO. For non-browser applications, we'll have to figure out what to do, but we can probably open a <window> with a <browser> element that points to the content, so that should be OK. The first version of the redesigned QAC will be Firefox specific, then we'll work on ports.
The events and news items will also have scroll-bars so that their content space if fixed but you can still read the items in that content space.
The Litmus Tab
This is by far everyone's favorite tab. It is difficult, complex and often overflows off people's screens. So, we need to fix it. The first thing we are going to do is remove the "list of tests" box. That will free up quite a bit of screen real estate. The end design we are thinking of will look something like:
====================================== | | | | Q | 'Litmus' | Bugs | | | | | | Current Test Run Foo [Change] | | | | | [<] [>] Test: Bar 4/5 Test Baz | | -------------------------------- | | | 1. These are the test steps | | | | 2. They will go here | | | | 3. And they will have a | | | | scrollbar | | | |______________________________| | | | | ------------------------------- | | | Expected Results will go | | | | here and will have a scroll | | | | bar too. | | | |_____________________________| | | | | X Passed Comment (optional): | | _ Failed | | | | _ Unclear |_________________| | | [SUBMIT RESULT] | |_____________________________________| |_________________________________[?]_|For comparison, the old dialog window:
So, when you click on the Litmus tab for the first time ever, you will be presented with a floating dialog over the tab that allows you to choose a test run to run from. This will be the same floating dialog that you will get if you click the [Change] button to change the test run and it will probably look a lot like the one you currently get except that it will hover over the center of the Litmus tab rather than being a true new window.
Once you have selected a test run, the next time you log in and go to the Litmus tab, you will automatically be presented with your last state. This includes if you were last on test X and you restarted the browser, then when you come back and log in, test X will be displayed once more with your comment replaced. It's session restore for litmus tests :). The Rest of the dialog is pretty self explanatory. The [<] and [>] buttons allow you to skip backward and forward through the tests in your current selected run.
What happens if the user gets to the end of that selected run? How would we handle an "random test" type of scenario? That is entirely up to you.
The Bugzilla Tab
And now the Bugzilla tab. So, if you declare a test as failed, we are thinking of showing you another of these floating dialogs and asking you to submit a bug. And if you say "yes, I'll submit a bug", then you come to this nifty Bugzilla tab.
The current Bugzilla tab does three things:
- Search for bug by ID
- Search for bug by text.
- Show selected bug X in Bugzilla.
New users do not really need to search Bugzilla. Finding duplicate bugs is not something that users who are new to the project are good at doing and it is a very high barrier to getting involved. All too often bugs do not get submitted because we tell people that they have to first search for dupes to their bugs, or the QA team has to help search for dupes during a test day. To address that, we are going to retool the entire thinking of the Bugzilla tab.
We want the new and improved Bugzilla tab help people file that first bug. So it needs to:
- Log you into Bugzilla.
- Provide the easiest possible interface to filing a bug.
- Automatically do a very simple search for dupes and show possible duplicates to this bug that is being filed.
Let's talk about the bug that the tool will enter first. We don't have much screen real estate and we don't want to duplicate Bugzilla. So, we will pre-fill a bunch of the fields for the user. We will fill in the OS, Platform, Component and Whiteboard fields for the user. We will put all the "QAC generated" bugs into Firefox/General with a whiteboard flag of "[QAC triage me]" and they will be filed as Unconfirmed bugs. Actually, we discovered that there is no way to add a whiteboard flag to the bug entry through bugzilla's RPC interface. So we will hve to appened this QAC tag to the summary. This will be transparent to the user. After the user enters their summary, we will prepend [QAC Generated] to the summary. This way, the QA team can easily query for any QAC filed bugs after a test-day and triage them appropriately.
The user will type in a Summary and a Description field for the bug. We'll prepopulate the description with some helper text to get them to write a decent report, and we will append the link to the Litmus test case if the bug is being filed from a test case that was marked "failed". One thing to consider is that if we are filing from a failed litmus test case we may not need to ask for things like "steps to reproduce" because those steps are in the litmus test case itself.
Once the user inputs a bug, we'll show a quick float over dialog (or change the main UI) that says "success" and gives them the link to the bug report, which they can then click to load into Bugzilla.
Now, the duplicate search ability is going to be very basic. Essentially if a new user is filing a bug from a failure in a litmus test case, then chances are good that this bug is already filed. So, we will check for duplicates using the following query:
- take the summary, query for summaries with "any of these words"
- in components Firefox/general, core/general (other high-landing areas that should be checked?)
- that have been filed in the last four weeks.
We will take a set of those bugs and bring up an info bar or some such UI to ask the user if his bug matches one of these existing ones. If the user clicks on one of the bugs then a Bugzilla tab will be opened for that bug in Bugzilla. We need to think out the rest of the interaction here to advise people how to add more information to existing bugs - ideally, we would have a system to easily instruct the user to leave a comment on said bug with the Litmus test case URL that we supplied them.
Therefore, we propose to do away with all searching in the Bugzilla tab. If you find the QAC Bugzilla searching useful, please let us know why and how, and we can reconsider. But at this time, we are thinking that the searching is simply too basic to be useful, and it is far more useful to provide a simple way for people to file bugs that they see in the product with the fewest number of hurdles in doing so.
Therefore, the proposed Bugzilla tab UI looks something like this:
====================================== | | | | Q | Litmus | 'Bugs' | | | | | Bug Summary: | | -------------------------------- | | | This is the summary | | | |______________________________| | | | | Bug Description: | | ------------------------------- | | | = Steps to Reproduce = | | | | 1. | | | | 2. | | | | 3. | | | | = Expected Behavior = | | | | | | | | = Actual Behavior = | | | | | | | | Litmus test: foo | | | | http://litmus.mozilla.org/foo| | | |_____________________________| | | | | [ENTER BUG] | |_____________________________________|For comparison, the old dialog window:
Other Thoughts / Moving Forward
The tab names should be what people are doing and not the web sites that they are using for it. So Litmus should probably become "Run Test" and Bugzilla should become "Enter Bug", so some such. The OS/Platform stuff should be guessed, but should be configurable in the QAC tab of the preferences pane, along with the notification settings and any other preferences we need. The "Logout" button will also be located there, and QAC will simply keep you logged in to Litmus until its cookie expires, which is what it does now.
Moving Forward
So, we encourage feedback and debate about all these ideas, nothing here is final, but we are going to start on the re-work of the UI and the backend changes needed to support these UI changes now. So, if you have thoughts get them in soon (https://wiki.mozilla.org/TDAI/QAC_Sprints QAC Sprints).
Up next, I want to propose a schedule for getting each of these tabs done in parallel sprints of code. So stay engaged and stay tuned, and let us know what you think.
Contacts
- Clint Talbert (ctalbert)
- Heather Arthur (harthur)
- Aaron Train (aaronmt)
