Steward

Anthony Hughes, Marcia Knous, Juan Becerra and Naoki Hirata

QA Contribute Group

Meetings every other Wednesday at 4 pm pacific.

• Meeting details at: QA/QA_Community#Meetings
• Notes from previous meetings

Action Plan

Note: QA community planning is also happening at https://wiki.mozilla.org/QA/QA_Community and https://wiki.mozilla.org/QA/Goals/2012Community

Goals

Pull together key goals from Anthony's metrics discussion with team.

Potential Contributors

Optimize Existing Channels

Audit existing channels where potential contributors find information (or should be able to find information) about how to get involved and then optimize to make sure the channels are effective.

• https://quality.mozilla.org

• http://nightly.mozilla.org and http://www.mozilla.org/projects/firefox/prerelease.html

• http://www.mozilla.org/firefox/aurora/ and Aurora First Run page?

• http://www.mozilla.org/firefox/beta/ and Beta First Run page?

• http://bugzilla.mozilla.org

• irc://irc.mozilla.org

• others???

Notes: get traffic stats for each to prioritize?

Retire Legacy Channels

For an area of the project that has been active for so long, such as QA, there will be many old pages with out of date information on them. For instance, a search for both 'mozilla testing' and 'mozilla qa' on Google brings up this page on the www-archive.mozilla.org site as the first result and QMO is not in the top set of results.

• Retire and redirect any legacy pages in top results for a Google search of 'mozilla testing' and 'mozilla qa'

Match Volunteers with Opportunities

• Create a list of role descriptions so people can easily understand what is expected of them? This might be something that is more appropriate for already active contributors looking to get more involved than for people just starting out.

• Create processes for manually connecting people with opportunities (eg, helping someone in a Test Day make a contribution) and for giving people a self-service way to get startted (eg, by installing the Nightly and following steps that show up on the first run page).

Active Contributors

Have a plan for getting newly active contributors into the phonebook and use relevant tags (ex, qa, automation, aurora, etc). This will allow us to reach out to experienced contributors with specific opportunities as they come up (for instance, you could email everyone with a 'aurora' tag if you were looking for help with a crasher that needs to be looked into by an experienced community member). We may want to document a set of tags we'd like people to use.

Have a process for taking someone who has come in through a QA entry point, such as installing the nightly, and introducing them to the full range of QA opportunities (desktop, web, automation, etc). It probably makes sense to hold off on presenting the full range of information until people are somewhat familiar with things instead of having this be something people see right away.

Create a dashboard that tracks relevant metrics and create a process for using those metrics. For instance, have a plan to reach out to any previously active contributor that hasn't been involved in the last two months to learn why they left or help them get involved again.

Core Contributors

Audit and update the existing module ownership information for QA modules. Identify process for recognizing new contributors and creating new modules/owners/peers.

Other Ideas To Consider

• Data and recommendations from recent QA community survey

• membership drive to get contributors into phonebook -- can use qa-branded swag for incentive (for instance, marcia knows that gabby is a contributor in argentina but where else are contributors?)

• redesign the nightly first run page and nightly.mozilla.org to make next steps clearer

• create a qa dashboard with metrics

• publish results from anthony's surveys -- learnings and next steps. for example, find out why people are reporting having a bad experience.

• optimize the experience for people sending inquiries about wanting to get involved with qa

• tap into ReMo program and work with reps

• events -- focus on upcoming latam mozcamp with a qa day?, attending she's geeky, etc.

• videos from contributors about how to get involved

• bugzilla process changes -- how to help someone filing their first bugs or opening bugzilla account? have a welcome to bugzilla email sent out to people who create new accounts with instructions on using it?

• providing tablets, hardware, etc. to contributors like community giving used to do

• marcia's brainstorming page: https://etherpad.mozilla.org/QA-Community-Brainstorm

Background Information

Identify Community

The following survey should help answer some questions: QMO blog post

Q: Can you identify all of the contributors on your team (both paid-staff and volunteer-staff)?

A:

• (ashughes) At a very high level, I probably could name 10 people who have participated in a testday and a couple who have attempted to write patches for Mozmill tests -- but to clearly identify ALL contributors and their contributions is not doable.
• (juanb) I cannot identify all of the volunteer contributors. I can identify a couple of volunteers who are very consistent, but not the occasional ones.
• (nhirata) I cannot identify all that are active contributors on the pay staff, as there are internal staff that are not a part of the team that have been contributing such as dria, asa, dburns, lmandel, and cbloom as can be seen in the bugzilla reporter column. There are some regular volunteers that help out with test day consistently that I have seen from week to week that try to help out. I would name gaby2300 as a star contributor in an effort to try to help; and she has been trying to lead efforts in being a QA community leader for her area. steffen.wilberg also helps out on his spare time (outside of test day)
• (marcia) Would be difficult to identify all of the volunteer contributors, especially if you count people who take the time to file bugs. I can identify a set of people I know who are working in areas related to QA.

Suggestion: Use the mozillians.org contributor directory to help. Communicate through your team's channels and encourage people to sign up and group themselves with a common team tag. If you assign a group tag to all contributors on your project, the Mozillians dashboard will track the size of that group and will also allow you to easily export the contact information for group members. You can export these contacts to ensure all your contributors are signed up.

Define Contribution Opportunities

Q: Can you point someone interested in contributing to your project to a list of available contribution opportunities?

A:

• (ashughes) I could probably rhyme off some contribution opportunities on request, but I could not point someone to a single page listing them
• (juanb) Today I cannot point them to a well defined list of contribution opportunities. It is one of our quarterly goals to produce a list of, say, 3 well defined tasks people can try, per functional QA team.
• (nhirata) https://wiki.mozilla.org/Mobile/Get_Involved; https://wiki.mozilla.org/Mobile (get involved section); in terms of testing mobile, mobile newsletter : https://mail.mozilla.org/listinfo/android-mobile-nightly-testers, https://wiki.mozilla.org/Mobile/Testdrivers_Program
  • to note : the mobile newsletter is syndicated in my blog, my twitter account, my people account, my facebook account, Planet, QMO, Browser Tech and Mobile blogs.
  • addendum : Scoobidiver has been very helpful to crashkill
• (marcia) The best summary we have right now is on QMO, but needs to be further refined. We need to chunk down some of the opportunities into smaller bites and we are working on that. Also, my "Contribute to Mozilla" initiative will work in that direction so that contributors can have direct contact with people who work in the various QA areas.

Suggestion: Look at what your team's needs are and what gaps you have in staffing to come up with a list of contribution opportunities. Capture those on a wiki page, in bugs, as role descriptions in Jobvite or whatever makes sense for your community.

Map Contribution Paths

Q: Are there clearly understood steps someone can follow to go from knowing nothing about your project to successfully contributing?

A: Here is the path for testing desktop Firefox:

• Download a nightly build

• Attend a test day

• ...

• (ashughes) Not really. I could personally help someone go from A to B with my knowledge and experience but I could not point them to any on-boarding documentation.
• (juanb) We haven't defined contribution paths. We tried coming up with these once, about two years ago.
• (nhirata) : I think the bar could be lowered for mobile; we do have contributor pages that are listed in answer to the last question.
• (marcia) I don't think right now we have a defined set, and it definitely would be team specific. Some teams do have better documentation - an example might be https://quality.mozilla.org/docs/webqa/projects/sumo/ which mentions options, but doesn't do a deep dive into the step by step.

Suggestion: In addition to just documenting these steps, look for a simple 5-minute task that someone can take to get started (for example, signing up for Bugzilla if they are interested in coding) and also figure out where in the process you can add a mentor to help people.

Establish Goals and Metrics

Q: Can you measure participation or contributors today? If so, what metrics can you track? What goal or metric would you like to achieve for Q1? Alternatively, what metrics would you like to get in place for Q1?

A: Anthony is leading a discussion with the QA team about what metrics they would like to get in place for Q1.

• (ashughes) We don't have a clear mapping of contributors, contributions, and metrics; nor any way to visualize that data if we had it. Our current priority is internally deciding what those metrics are.
• (juanb) We can measure participation in test days, and we can measure emails from people interested in participating. We don't have the on-ramps in place so we can get them to try something yet.
• (nhirata) We could potentially measure the bug count/bug verifications from MTD as well as testdays, and participation in test days. I am uncertain if that does give a clear mapping of contributors, contributions and metrics.
  • addendum: also having a search in litmus for test cases or test runs. For webqa, potentially pushes to selenium tests may be checked.
• (marcia) Bugzilla probably would provide the most metrics, since many QA contributors file bugs. Next year I was thinking about having a membership drive - if we had something like that we track entry for newcomers. I also think another way to track contributors might be getting them to join the community phonebook and have a field for the date they joined the project.

Suggestion: Write down what you think would be helpful to track even if it isn't possible to get that data today. We'll work on implementing dashboards when we know what data we want.