QA/Desktop Firefox/Meetings:2012-07-10

Desktop QA Meeting

Chair:

Scribe:

Previous: https://wiki.mozilla.org/QA/Desktop_Firefox/Meetings:2012-07-02

This: https://wiki.mozilla.org/QA/Desktop_Firefox/Meetings:2012-07-10

Attendees:

* 

Previous Actions (add status update here):

If the status item is not complete, include it as an action item for the following week

Anthony

* [IN PROGRESS] Find out Q3 plans for MozTrap

Jason

* [IN PROGRESS] Get an update on persona integration for Q3 in desktop

Juan

* None

Marcia

* Build the text for postcard for desktop QA

Tracy

* move 10-12 Sync test cases over to Moztrap
* add buddy notes to https://wiki.mozilla.org/QA/Desktop_Firefox/BuddySystem

Agenda

* Planning Q3 Goals - Scoping, Themes, and Beginning to Formalize
  * Themes from Ideas
  * Scope - Critical, Non-Critical, Out of Scope
  * Formalization
* Buddy System - https://wiki.mozilla.org/QA/Desktop_Firefox/BuddySystem

Open Questions

* (ashughes) I think we need a goal or subgoal around Desktop Performance, perhaps consult Engineering goals, given it is a top-3 company priority for Q3
  * I've been asked by Engineering to organize a testday to test the top-30 non-AMO add-ons for memory leaks this quarter; perhaps that fits in here
  * (jsmith) Right, heard that in the MoCo meeting as well. Something in regards to Snappy? Memshrink?
  * (jsmith) Another area I've always wondered - should we be involved in the teams (and I mean directly, meaning in the meetings)
  * Anthony - Followup Lawrence and Taras on Desktop Snappy Goals for Q3 - How does it link to Desktop QA?
* How should triaging and verifying k9o and basecamp fixes tie into our goals?
  * (jsmith) QA org wide goal - Desktop should have a sub-bullet item involving this. Following up with Bob about this.
  * (jsmith)   We need to help when help is needed - probably best approach is to  have  a QA org goal for verification rate on qa+ and our goal is  supporting  it when needed
  * (ashughes) +1 - I think we need something around triage and basecamp requirements
  * Incorporate Basecamp & k9o bug metrics into beta release process and meet it
  * (jsmith)  This is now scoped down to basecamp, as k9o project is dead. Basecamp  bugs aren't usually desktop either (they can be, but its likely to be rare). What should we do here?
  * (jsmith) I know for me at least, I'm stuck testing the Apps stuff on B2G (install flow, permissions) until someone can take over (I've been delegated as the owner). The mobile web compat stuff happens on the side, but I've also considered an owner here. Puzzled how these get called out. Do we not call them out and de-scope our goals to give time to support mobile goals more? 
  * (jsmith) Followup with Bob and Stephen Donner on how to handle this

Resolved Questions - Finished with Possible Goals Included

* What work is targeted for sign into browser or native persona integration for Q3?
  * (jsmith) Emailed Dan Mills about this
  * (ashughes) +1 - whats the status/scope of development?
  * (jsmith) Followup dan mills again <-- jason
  * (jsmith)  based on what I heard into today's weekly updates meeting, there's  pieces landing, so I think that "native persona integration" support  might be in-scope
  * (jsmith) Emailed John N. to see if he knows. Dolske is driving this.
* Web API - What's in scope for desktop? What basecamp items could go here that desktop could support? Automation?
  * (jsmith)   Geo is driving this. No plans known right now on the desktop side,   although if there is an end-user facing version of the API that comes   down the pipeline, then we should focus on it (camera API). But right   now I've got no knowledge of this happening.
    * (ashughes)  based on that I think we leave it off the goals but need to keep  revisiting it with Geo (maybe every two weeks) and be flexible to add it  to the goals as needed
* What work is targeted for the Social API for Q3 for Desktop Firefox?
  * (jsmith) P1 for Desktop Firefox 16 - Probably needs to be a part of our goals
  * (ashughes) Definitely in scope and on my plate, thinking something like...
    * "Qualify Social integration with Firefox 16 in collaboration with our launch partner"
* Web RTC - What's in scope for Q3? Camera API?
  * (jsmith)   Confirmed that we need provide QA support for this in Q3. Heavily more   on the Camera API side for testing up to FF Beta, but ramping up and   establishing a plan for WebRTC as a whole.
    * (ashughes)  What Firefox version is CameraAPI scope for? We could certainly have a  goal around WebRTC whereby "phase 1" is delivery of the Camera API
* What apps presentation features are in scope for Q3?
  * (jsmith)   Emailed Ragavan and Jen about this. Confirmed today this is definitely   in scope for FF 16 as an add-on through the newnewtab project.
    * (ashughes) what bucket does this fall into? basecamp, k9o, WebApps?
* What Q3 goals could we pull from the community action plan? Contributor path?
  * (jsmith) Defining and implementing a contributor path for three contributors?
  * (ashughes) review, revise, roadmap, and milestone; then deliver on Milestone 1?
  * Define roadmap & deliver on first milestone of roadmap
* What followup goal could we derive for the contributor mailing list?
  * (jsmith) Some rate of response is probably a good goal like what we did for Q2
  * (ashughes) On-board one member from contribute@ to a project contribution?
  * Goal should be around number of contributors on-boarded from mailing list
  * One contributor is critical, 3 is stretch?
* Should we keep the test day community dashboard in-scope for Q3 as a goal?
  * (jsmith) Something smaller scale is probably okay (bugzilla wikimo plugin), but something fancy should be out of scope
  * (ashughes)   I need to talk with Metrics about their Q3 priorities but perhaps it   would be good to form a goal around something that is not dependent on   them I've since followed up with   Metrics and it's  been deprioritized because it's not critical to   Company goals (working  with dboswell to figure out next steps)
    * We   have a bot/email which sends me IRC nicks and messages (this is done   manually by David Burns) -- we could strive to automate this in Q3
    * I also have a spreadsheet which tracks # active testers and # bugs tagged, perhaps we can strive to automate this too
    * My   vision is something like the Code Contributor dashboard but for tests   run and bugs filed/triaged, we might be able to extend/reuse that
    * stretch goal, non-dependent on metrics
* Should we continue to work on find:fix rate for crashers? Or do something else entirely?
  * (jsmith) Marcia could probably answer this. Should be reflective of what crashkill intends to for Q3 for process improvement.
    * (ashughes) +1
  * Be reflective of crashkill process improvement
  * Goal - Expand our testing capabilities to start analyzing top crashers for desktop runtime
* How should mobile layout compatibility tie into our goals?
  * (jsmith)   Depends on how we handle adding basecamp goals not directly tied to   desktop into desktop. Following up with Bob about this
  * (jsmith) Technically not directly tied to desktop, but it is tied to apps - should QA mobile make this a goal then, not us?
  * (ashughes) Goals with external dependencies typically fail.
    * We should definitely wrap goals around our team's requirements for Basecamp
    * If Mobile team needs our help, we should help, but that doesn't need to be tied to a goal
  * Not tied to desktop - this is a mobile goal
* Are there any process changes we could make to utilize the QA contact field?
  * (jsmith) Maybe something in relation to explicit ownership. Maybe defining the process and implementing it?
  * (ashughes) +1 - this was returned to us on July 4
    * https://bugzilla.mozilla.org/show_bug.cgi?id=684088
    * Goal   might be to develop a process for "owernship" and to have all P1   components identified and owned (some by community) by the end of Q3
    * Note that there has to be some shared strategy here as it affects the entire QA org but the Desktop QA team can be the leader
  * Defining and implementing the process for QA contact in Q3 <-- Goal
* (ashughes) Are we aiming to have one brownbag or seeking to imrpove that to two?
  * One for Q3
* (ashughes) In Q2 we missed our goal to automate some WebApps tests, what are we scoping for Q3?
  * Scoped for Q3
* (ashughes) What's next for the Buddy System
  * Define and implement buddy system process <-- Goal
* What work needs to be done for the Italy Event for Q3?
  * (jsmith)   Marcia could probably answer this. Should this be tracked as a desktop   goal or a QA Community wide goal? Sounds more like a QA Community wide   goal.
    * (ashughes)   There's likely a QA-team goal with a QA-desktop contribution goal (ie.   just because it should be a top-leve QA-org priority, does not mean  our  team can't have a contributing goal)
  * (ashughes) What other events are either on our radar, or should be?
  * Refining the plan for Q3 Italy Event - Goal?
  * (jsmith) Perhaps this should be a QA community goal
  * http://gracehopper.org/2012/ will occur Q4 but planning in Q3
  * MozCamp EU?
  * (jsmith)  All of these sound great, but I think these all fall under QA  Community, not desktop. There might be things we do that help these  though.
  * marcia to add this to QA community goals

Resolved Questions - Needs a Goal

* What work should be targeted for unconfirmed bug management improvement?
  * (jsmith) Probably should aim for some rate of confirmation in relation to the number of unconfirmed bugs that show up in Q3
    * (ashughes) +1
  * Improve unconfirmed bug confirmation rate for Q3 bugs filed by x%?
  * Participation as another goal
  * Goal - Formulization of bugmaster continuous days, get involvement for it
  * Anthony - followup in creating goal for this
    * Could be opportunity to use MozEU Camp to help with this goal
* What forms of chemspill process improvement strategies can we do?
  * (jsmith)  Brought up by the guys driving basecamp - we take control of this, as  we need to limit pulling resources from other teams if possible and  knowing how to coordinate well ourselves if possible
    * (ashughes) two things that come to mind:
      * automating tests for the things we regressed
      * improving ability to react to late-night, weekend demands
  * Goal - Coming up with strategy for chemspill management from start to end and implement it
  * MozTrap & automation for chemspills - process and execution
  * Jason writeup strategy goal - should include automation pieces
* (ashughes) Do we want a goal around making our day-to-day more publicly accessible?
  * meetings, triage, release testing, feature testing, etc
  * Goal - Regular Blog Posts About Things to Do in Desktop and Getting Involved and Other Stuff?
  * Non-Critical
  * Jason to write up goal
* (ashughes) Do we want a goal around having a live testday / meetup style event this quarter?
  * Test Day Live - Marionette on Desktop
  * File a bug day!
  * Anthony to look into
* (ashughes) Do we need a goal for integrating OSX 10.8 as a supported platform?
  * Qualify OS X 10.8 latest FF releases
  * Anthony to look into
* (ashughes) Do we need a goal for ramping up Win 8 integration testing?
  * Planning piece for testing <-- Q3
  * Anthony to look into
* (ashughes) We need a goal around continuing to ramp up Rapid Betas
  * "Develop and execute on a test strategy resulting in delivery of Betas on a (semi)daily basis
  * Anthony to look into
* (ashughes) What's next for MozTrap? Do we want a goal around retiring Litmus?
  * Dependent on ateam priorities <-- followup ateam
  * Anthony is going to followup
  * Add goal - Integrating automation results into MozTrap, retire Litmus 100%
  * Anthony to look into

Formalization

* Qualify the Mozilla Web Platform
  * [Critical] Qualify the WebRT Desktop feature
  * [Non-Critical] Qualify the AITC Desktop feature
  * [Critical] Qualify the Social API feature
  * [Critical] Qualify the Camera API feature
  * [Non-Critical] Qualify native persona integration into Desktop Firefox
    * (jsmith) Dependent on response from Dolske
  * [Non-Critical] Qualify the New Tab Integration feature for Web Apps on Desktop
    * (jsmith) Need to followup with Todd, Ed, Bill, Asa
  * [Non-Critical] Develop a high quality test plan for the full stack of WebRTC on desktop
    * (jsmith) Dependent on conversation with Maire and Randell
  * [Critical] Implement automated coverage for the smoke tests for the apps lifecycle for desktop
    * (jsmith) Followup with Clint and David Burns
* Improve retention of Desktop QA community membership
  * [Critical] Define roadmap and deliver on first milestone of community roadmap
  * [Critical] On-board at least one contributor towards owning a bugzilla component
  * [Non-Critical] Deliver a dashboard to visualize test day involvement
* Improve visibility and accessibility of the Desktop QA team
  * [Critical] Deliver one brownbag to the organization and community
* Improve effectiveness and efficiency of the release qualification process
  * [Critical] Start analyzing top crashers top crashers for desktop web apps
    * Dependency: Volume of users using web apps on desktop (750,000 users end of August)
  * [Critical] Define and implement the process for tracking QA ownership of bugs
  * [Critical] Define and implement a buddy system for knowledge sharing and responsibility redundancy

Action Items:

Anthony

* 

Jason

* 

Juan

* 

Marcia

* 

Tracy

*