QA/Desktop Firefox/Meetings:2012-07-17
From MozillaWiki
< QA | Desktop Firefox
Desktop QA Meeting
Chair: Scribe:
Previous: https://wiki.mozilla.org/QA/Desktop_Firefox/Meetings:2012-07-13 This: https://wiki.mozilla.org/QA/Desktop_Firefox/Meetings:2012-07-17
Attendees:
*
Previous Actions (add status update here):
If the status item is not complete, include it as an action item for the following week
Anthony
* None
Jason
* [IN PROGRESS] Get an update on persona integration for Q3 in desktop
Juan
* None
Marcia
* Build the text for postcard for desktop QA
Tracy
* move 10-12 Sync test cases over to Moztrap * add buddy notes to https://wiki.mozilla.org/QA/Desktop_Firefox/BuddySystem
Agenda
* Finalize first draft of Q3 goals for desktop qa
Resolved Questions - Finished with Possible Goals Included
* What work is targeted for sign into browser or native persona integration for Q3?
* (jsmith) Emailed Dan Mills about this
* (ashughes) +1 - whats the status/scope of development?
* (jsmith) Followup dan mills again <-- jason
* (jsmith) based on what I heard into today's weekly updates meeting, there's pieces landing, so I think that "native persona integration" support might be in-scope
* (jsmith) Emailed John N. to see if he knows. Dolske is driving this.
* Web API - What's in scope for desktop? What basecamp items could go here that desktop could support? Automation?
* (jsmith) Geo is driving this. No plans known right now on the desktop side, although if there is an end-user facing version of the API that comes down the pipeline, then we should focus on it (camera API). But right now I've got no knowledge of this happening.
* (ashughes) based on that I think we leave it off the goals but need to keep revisiting it with Geo (maybe every two weeks) and be flexible to add it to the goals as needed
* What work is targeted for the Social API for Q3 for Desktop Firefox?
* (jsmith) P1 for Desktop Firefox 16 - Probably needs to be a part of our goals
* (ashughes) Definitely in scope and on my plate, thinking something like...
* "Qualify Social integration with Firefox 16 in collaboration with our launch partner"
* Web RTC - What's in scope for Q3? Camera API?
* (jsmith) Confirmed that we need provide QA support for this in Q3. Heavily more on the Camera API side for testing up to FF Beta, but ramping up and establishing a plan for WebRTC as a whole.
* (ashughes) What Firefox version is CameraAPI scope for? We could certainly have a goal around WebRTC whereby "phase 1" is delivery of the Camera API
* What apps presentation features are in scope for Q3?
* (jsmith) Emailed Ragavan and Jen about this. Confirmed today this is definitely in scope for FF 16 as an add-on through the newnewtab project.
* (ashughes) what bucket does this fall into? basecamp, k9o, WebApps?
* What Q3 goals could we pull from the community action plan? Contributor path?
* (jsmith) Defining and implementing a contributor path for three contributors?
* (ashughes) review, revise, roadmap, and milestone; then deliver on Milestone 1?
* Define roadmap & deliver on first milestone of roadmap
* What followup goal could we derive for the contributor mailing list?
* (jsmith) Some rate of response is probably a good goal like what we did for Q2
* (ashughes) On-board one member from contribute@ to a project contribution?
* Goal should be around number of contributors on-boarded from mailing list
* One contributor is critical, 3 is stretch?
* Should we keep the test day community dashboard in-scope for Q3 as a goal?
* (jsmith) Something smaller scale is probably okay (bugzilla wikimo plugin), but something fancy should be out of scope
* (ashughes) I need to talk with Metrics about their Q3 priorities but perhaps it would be good to form a goal around something that is not dependent on them I've since followed up with Metrics and it's been deprioritized because it's not critical to Company goals (working with dboswell to figure out next steps)
* We have a bot/email which sends me IRC nicks and messages (this is done manually by David Burns) -- we could strive to automate this in Q3
* I also have a spreadsheet which tracks # active testers and # bugs tagged, perhaps we can strive to automate this too
* My vision is something like the Code Contributor dashboard but for tests run and bugs filed/triaged, we might be able to extend/reuse that
* stretch goal, non-dependent on metrics
* Should we continue to work on find:fix rate for crashers? Or do something else entirely?
* (jsmith) Marcia could probably answer this. Should be reflective of what crashkill intends to for Q3 for process improvement.
* (ashughes) +1
* Be reflective of crashkill process improvement
* Goal - Expand our testing capabilities to start analyzing top crashers for desktop runtime
* How should mobile layout compatibility tie into our goals?
* (jsmith) Depends on how we handle adding basecamp goals not directly tied to desktop into desktop. Following up with Bob about this
* (jsmith) Technically not directly tied to desktop, but it is tied to apps - should QA mobile make this a goal then, not us?
* (ashughes) Goals with external dependencies typically fail.
* We should definitely wrap goals around our team's requirements for Basecamp
* If Mobile team needs our help, we should help, but that doesn't need to be tied to a goal
* Not tied to desktop - this is a mobile goal
* Are there any process changes we could make to utilize the QA contact field?
* (jsmith) Maybe something in relation to explicit ownership. Maybe defining the process and implementing it?
* (ashughes) +1 - this was returned to us on July 4
* https://bugzilla.mozilla.org/show_bug.cgi?id=684088
* Goal might be to develop a process for "owernship" and to have all P1 components identified and owned (some by community) by the end of Q3
* Note that there has to be some shared strategy here as it affects the entire QA org but the Desktop QA team can be the leader
* Defining and implementing the process for QA contact in Q3 <-- Goal
* (ashughes) Are we aiming to have one brownbag or seeking to imrpove that to two?
* One for Q3
* (ashughes) In Q2 we missed our goal to automate some WebApps tests, what are we scoping for Q3?
* Scoped for Q3
* (ashughes) What's next for the Buddy System
* Define and implement buddy system process <-- Goal
* What work needs to be done for the Italy Event for Q3?
* (jsmith) Marcia could probably answer this. Should this be tracked as a desktop goal or a QA Community wide goal? Sounds more like a QA Community wide goal.
* (ashughes) There's likely a QA-team goal with a QA-desktop contribution goal (ie. just because it should be a top-leve QA-org priority, does not mean our team can't have a contributing goal)
* (ashughes) What other events are either on our radar, or should be?
* Refining the plan for Q3 Italy Event - Goal?
* (jsmith) Perhaps this should be a QA community goal
* http://gracehopper.org/2012/ will occur Q4 but planning in Q3
* MozCamp EU?
* (jsmith) All of these sound great, but I think these all fall under QA Community, not desktop. There might be things we do that help these though.
* marcia to add this to QA community goals
* (ashughes) Do we want a goal around having a live testday / meetup style event this quarter?
* Test Day Live - Marionette on Desktop
* File a bug day!
* Anthony to look into
* (ashughes) Do we need a goal for integrating OSX 10.8 as a supported platform?
* Qualify OS X 10.8 latest FF releases
* Anthony to look into
* (ashughes) Do we need a goal for ramping up Win 8 integration testing?
* Planning piece for testing <-- Q3
* Anthony to look into
* (ashughes) We need a goal around continuing to ramp up Rapid Betas
* "Develop and execute on a test strategy resulting in delivery of Betas on a (semi)daily basis
* Anthony to look into
* (ashughes) What's next for MozTrap? Do we want a goal around retiring Litmus?
* Dependent on ateam priorities <-- followup ateam
* Anthony is going to followup
* Add goal - Integrating automation results into MozTrap, retire Litmus 100%
* Anthony to look into
* What work should be targeted for unconfirmed bug management improvement?
* (jsmith) Probably should aim for some rate of confirmation in relation to the number of unconfirmed bugs that show up in Q3
* (ashughes) +1
* Improve unconfirmed bug confirmation rate for Q3 bugs filed by x%?
* Participation as another goal
* Goal - Formulization of bugmaster continuous days, get involvement for it
* Anthony - followup in creating goal for this
* Could be opportunity to use MozEU Camp to help with this goal
* What forms of chemspill process improvement strategies can we do?
* (jsmith) Brought up by the guys driving basecamp - we take control of this, as we need to limit pulling resources from other teams if possible and knowing how to coordinate well ourselves if possible
* (ashughes) two things that come to mind:
* automating tests for the things we regressed
* improving ability to react to late-night, weekend demands
* Goal - Coming up with strategy for chemspill management from start to end and implement it
* MozTrap & automation for chemspills - process and execution
* Jason writeup strategy goal - should include automation pieces
* General strategy I've envisioned for automation is as follows:
* 1. Identify the "chemspill" triggers/bugs
* 2. Identify which of those triggers do not have in-testsuite+
* 3. Write MozTrap smoketests to cover those scenarios for future releases
* 4. Write Mozmill tests to cover the MozTrap regression tests
* 5. Remove the MozTrap tests which are automated
* (ashughes) Do we want a goal around making our day-to-day more publicly accessible?
* meetings, triage, release testing, feature testing, etc
* Goal - Regular Blog Posts About Things to Do in Desktop and Getting Involved and Other Stuff?
* Non-Critical
* Jason to write up goal
* (ashughes) I think we need a goal or subgoal around Desktop Performance, perhaps consult Engineering goals, given it is a top-3 company priority for Q3
* I've been asked by Engineering to organize a testday to test the top-30 non-AMO add-ons for memory leaks this quarter; perhaps that fits in here
* (jsmith) Right, heard that in the MoCo meeting as well. Something in regards to Snappy? Memshrink?
* (jsmith) Another area I've always wondered - should we be involved in the teams (and I mean directly, meaning in the meetings)
* Anthony - Followup Lawrence and Taras on Desktop Snappy Goals for Q3 - How does it link to Desktop QA?
* How should triaging and verifying k9o and basecamp fixes tie into our goals?
* (jsmith) QA org wide goal - Desktop should have a sub-bullet item involving this. Following up with Bob about this.
* (jsmith) We need to help when help is needed - probably best approach is to have a QA org goal for verification rate on qa+ and our goal is supporting it when needed
* (ashughes) +1 - I think we need something around triage and basecamp requirements
* Incorporate Basecamp & k9o bug metrics into beta release process and meet it
* (jsmith) This is now scoped down to basecamp, as k9o project is dead. Basecamp bugs aren't usually desktop either (they can be, but its likely to be rare). What should we do here?
* (jsmith) I know for me at least, I'm stuck testing the Apps stuff on B2G (install flow, permissions) until someone can take over (I've been delegated as the owner). The mobile web compat stuff happens on the side, but I've also considered an owner here. Puzzled how these get called out. Do we not call them out and de-scope our goals to give time to support mobile goals more?
* (jsmith) Followup with Bob and Stephen Donner on how to handle this
* (jsmith) Talking with Stephen, our goals definitely need to call out Firefox OS items, even though they are not desktop-related. We also need to be cautious that they serve as an important piece of desktop goals, overtaking others that we might have already defined.
* (jsmith) The areas to call out for basecamp testing likely would be the areas I own at least - B2G web apps, mobile web compat support on B2G, and live test day for mobile web compat
Feedback
Bob's feedback in summary - high-level, sub-bullets could provide measurements. Should be something more along the lines of:
1. FF 15 & 16 Desktop & Mobile and Basecamp 2. Leverage (grow mozilla) - community involvement & automation on desktop & release process
Victory - runs on time, no chemspills by oversight by Mozilla
Goals Take Two
1. On-Time Qualification of Desktop Firefox 15 and 16 releases with no chemspills by oversight by the Mozilla Organization 2. Support Qualification of the Firefox OS basecamp milestone and Firefox for Android 15 and 16 releases as needed 3. Leverage the wider community and automation to provide extensive and efficient test coverage of Desktop Firefox releases
Old List of Goals (Ignore This)
* Qualify the Mozilla Web Platform and improve QA involvement with major platform initatives * [r+] [Critical] Qualify and provide direct involvement with the WebRT Desktop feature * [r+] [Critical] Qualify and provide direct involvement with the Social API feature * [r+] [Critical] Qualify and provide direct involvement with the Camera API feature * [r+] [Critical] Develop and implement a strategy to qualify Desktop Firefox performance improvements on a per release basis * [?] [Critical] Qualify and provide direct involvement with the Firefox OS Web Apps Feature (sync up with Stephen and Bob) * Improve retention of Desktop QA community membership * [r+] [Critical] On-board at least one contributor towards owning a bugzilla component * [r+] [Non-Critical] Improve transparency into Desktop QA by posting bi-weekly blog posts on what's currently going on and how to get involved * Improve effectiveness and efficiency of the release qualification process * [r+] [Critical] Define and implement the process for tracking QA ownership of bugs * [r+] [Critical] Define and implement a buddy system for knowledge sharing and responsibility redundancy * [r+] [Critical] Develop and Implement a Strategy for Managing Chemspill Releases and Past Chemspill Problems
Goals Cut
* [r+] [Critical] Start analyzing top crashers for desktop web apps
* Dependency: Volume of users using web apps on desktop
* Lead - Jason Smith
* [r-] [Critical] Qualify MacOSX 10.8 for integration as a supported platform in Firefox 15
* (jsmith) Do we need to explicitly call this out? We already expanded our testing capabilities to include OS X 10.8 in Q2. At this point, wouldn't we be doing our job with testing OS X 10.8?
* [r-] [Critical] Develop a strategy for qualification of Betas on a daily basis
* [r-] [Critical] Deliver on an automation strategy which includes smoketests, feature tests, chemspill regression tests, and onboarding feature owners as developers
* Improve visibility and accessibility of the Desktop QA team
* [r+] [Critical] Deliver one brownbag to the organization and community
* (jsmith) Already got a great idea for this - panel discussion to discuss about the evil mind of a QA!
* Lead - ?
* QA Overall Goal?
* [r-] [Critical] Define roadmap and deliver on first milestone of community roadmap
* (jsmith) As much as I like roadmaps, I think we should stick with managing our community approach on a per quarter basis in terms on desktop. We might want to make this a QA Community goal, as it feels more applicable to grand scheme of the QA team at large.
* Lead - ?
* [r-] [Critical] Implement automated coverage for the smoke tests for the apps lifecycle for desktop
* (jsmith) Followup with Clint and David Burns
* Lead - Jason Smith
* (jsmith) I think we should offset this to David for Q3 in the ateam's goals
* [r-] [Non-Critical] Qualify native persona integration into Desktop Firefox
* (jsmith) Dependent on response from Dolske
* (jsmith) This can have side benefits to B2G, as they touch similar areas. I do think identity pieces as Gary said was lower priority, so the priority appears to be correct here.
* (jsmith) I think we should cut this and rely on softvision coverage.
* Lead - Tracy Walker
* [r+] [Non-Critical] Qualify the AITC Desktop feature
* [?] [Critical] Qualify any other WebRTC landed pieces
* [?] [Non-Critical] Deliver a dashboard to visualize test day involvement
* [?] [Non-Critical] Increase the number of UNCONFIRMED bug triage contributors
* (jsmith) Not quantifiable. What is deemed as a sucess for this goal? Re-think this to make this quantifiable.
* [r-] [Non-Critical] Organize a "meetup" style event to teach people QA fundamentals (ie. filing a bug)
* (jsmith) Though I am in favor of a live test day, the basecamp priorities discussion I had makes me rethink what we should be doing here. The mobile web compat stuff might actually now apply here, given that each team does need to include explicitly called out B2G items in their goals. Maybe this should be in relation to mobile web compat for B2G (supporting it)?
* QA Community Goal
* [r+] [Critical] Qualify Desktop Firefox on Non-Metro Windows 8 in Firefox 16+
* [r+] [Non-critical] Integrate MozMill test results into MozTrap and retire Litmus
* Depends on A-Team for linking Mozmill results to MozTrap
Action Items:
Anthony
*
Jason
*
Juan
*
Marcia
*
Tracy
*
