QA/Community/Leadership

Community Leadership ideas from Meeting with Schrep, Timr, Jay Written up by Jay Mar 4, 2008

Here are some of the points I took away:

1. When thinking about how to measure community involvement or personal success, imagine the perfect tools exist and come up with what you would expect those tools to tell you. Then identify proxies that might give us some data and make sense of it the best you can.
2. While absolute numbers increase (total productivity), ratio of community/internal involvement should go up as well to maintain balance in contributions of employees vs volunteers (makes sure community continues to grow as internal resources grow). Notes from 10/14/08 mtg: Grow community as the employee's grow. Keep good balance between internal MoCo and Community volunteers. Everyone in QA team should be leveraging community members
3. Treat QA contributors like any other contributor, make sure they are represented for testing as much as developers and localizers (identify people for "Friends of the Tree", Firefox credits, invitations to summits, community team t-shirts, etc)
4. Don't worry about the input/output ratio, but focus on measuring incremental change and what it took to see that change. Sometimes will be positive, other times negative, but be prepared to show people what you put in, how you measured it, and focus on tweaking the process to make things better over time.
5. Target 4 areas (for example), and tell the story that will bring them all together. Show why we did something, how it influenced the team/project, and what we gained from it.
6. Try to take a scientific approach to community growth. Try things and measure their impact. Did Bug Days over a few months bring down the unconfirmed bug list down by 100? Did someone that logged crash bugs follow up on them with data or steps to reproduce; did this impact the stability of a future release? Will QMO increase overall test day participation 10% based on last year's numbers? Note: show off our accomplishments. Blog!
7. Understand both macro and micro metrics/decisions... iterating through micro changes will help develop a story to explain macro changes. Example he gave was through all the work with QMO, not only think about success of events/projects (micro), but see how that all adds up to developing quality contributors (macro). See how much work it might take to win over one new regular contributor... and be able to explain how that happened.
8. Don't be afraid to share small bits of data or findings on a regular basis. Don't need to wait to have a complete picture to share work and progress with the organization or community. Blog often, even if just a couple of paragraphs.

My thoughts on what a growing and healthy community will look like (realistically) are:

1. More regular visitors to QMO, both in terms of events and project participation, as well as forum discussion. No data available from the past year, since there as not been any form of measurement or actual tracking done, but some regular level of increasing community activity at the site will be a good sign of growth (which we will be able to start measuring from day 1 of the official launch)
2. Increasing level of feedback through surveys and/or mailing lists (betatesters, qa-community). This means seeing more people respond to our surveys or seeing more bugs/concerns reported from people on our mailing lists during specific test requests. We are already seeing this with our first QA community survey, and will get more specific feedback about Firefox 3 beta when we put out a new survey to the betatesters alias this month.
3. One or two projects being well run by QA team members, with at least 1 or 2 community members participating over a number of months. At least making steady progress (as with the unconfirmed bug triage initiative Sam started) during one or two quarters. Or producing something that is useful to the QA team and community (like the QA Companion) over the period of a year.
4. Converting at least 1 or 2 volunteers into regular contributors a year... which could include anything from someone becoming a regular Bugzilla watcher to identifying someone that can be hired on full-time. For those that don't convert, a sign of growth and health would be the ability to track a person's progress from signing up to be a QMO member to whatever they end up accomplishing while they are active members of the community (they may not be around forever, but learning how they grow and develop will be a huge plus). Note: regular participation, known by people on the QA team. Ex: Ria, Aleksej
5. Hearing from QA team members about someone that did something great for the project on a monthly basis will also be a sign of community success. Being able to keep a steady influx of new volunteers and giving them the tools necessary to accomplish great things one by one, we will be able to identify regular QA Community All-Stars, which surely is a sign of a growing, healthy community.
6. Improvements in certain volunteers' ability to log quality bugs, write test cases, or verify bugs will also be a sign of a healthy community. This will validate any work the QA team does with the contributor as they learn from mistakes, take more initiative, and get comfortable with others in the Mozilla community. I believe the "Badge" system for QMO will allow us to track this one way... there are probably other ways we can improve this as well.
7. Having the tools and data on a quarterly basis to present some sort of QA community report will be a good thing as well. Knowing exactly how many people were active in Bugzilla or Litmus each quarter will help us track growth.
8. All of the above will obviously be tied into both major and minor Firefox releases, so understanding how well contributors did vs the QA team in certain areas will also give us a good idea on how well we are growing. We need to figure out how we best measure this, as it's not as simple as what Schrep said is used for developers (frequency, size, and quality of patches landed). So we need to find out how we can translate community work into improved quality for our releases... but given that we don't do this for our team in general, it might be challenging.