QA/Testdays/Framework 2.0
This page tracks the development of a new framework for self-directed testdays. The intent is to have a document which facilitates individuals planning events to introduce contributors to their projects.
Contents
Phase 1: Consultation
The purpose of this phase is to consult with stakeholders to inform a new framework for self-directed testdays focused more on mentorship and connecting new contributors with projects.
1. Collect feedback from organizers [DONE]
- streaming the event via AirMozilla should be an integral part of every event
- limit events to teaching a single skill/tool
- limit events to two hours (or multiple 2hr sessions depending on host availability)
- start with a lesson via video conference, followed by Q&A (record and publish to AirMo following event)
- end events with instructions on how to carry that skill forward to on-going contribution
- target events at different times, not just 9-5 Friday in local timezone
- provide clear instructions/requirements but make sure everyone is included in time
- profile testday audience to get a clear picture of test environments available and who's showing up
- all events should be locally hosted and remotely accessible
- have some way to explain Mozilla's philosophy for doing this type of event (localized video perhaps?)
- connecting with a long-term mentor is important to contributor continuity
- create a list that attendees can sign up to to be re-invited to future events
- provide a list of opportunities inside and outside QA that they might contribute to
- conduct events and provide instruction in the local language as much as possible
- Reps should be a key participant in organizing events on the ground
- make sure no one is left behind due to learning curve
- intuitive tasks (eg, addons testing) are more fun out of the gate, more difficult tasks (eg. bug triage) require learning before they become fun
- all associated tasks need documentation and one&done activities
- be sure to get feedback from participants swiftly following an event, make sure you collect email addresses for follow-up
- some people are shy to engage directly - offer them the opportunity to ask questions, give feedback, and participate out-of-band
2. Collect feedback on brand [DONE]
- diversify the topic offering beyond testing and bug triage for Firefox/Fennec (engage with QA & Engineering teams to identify relevant topics)
- provide clearer/localized instruction and documentation for all testday topics
- build a framework for recognition as it relates to testdays but with an eye to recognition QA-wide (include recognition of work that does not yield an artifact like a bug)
- define the brand of testdays we want to put forward and repeatedly evangelize/showcase that brand
- integrate pathways for different projects but also for different roles/responsibilities and Mozillian levels
- have times in UTC on the test plans
- links to moderator profiles in Mozillians (make sure these are complete)
- don't have event if no one is around to moderate
- etherpad chat causes confusion and often goes unnoticed
- some people post in the etherpad days before the event - suggest pushing people to IRC to get the testplan and not publish the etherpad
- Google Chrome translation overwrites etherpad content
3. Collect feedback on recognition [DONE]
- develop a framework for building long-term relationships and ensuring people don't get lost/discouraged
- develop a framework for growing as a contributor across several differentiating projects
- develop a framework for proactive, personalized, and individualized recognition (when/where/how)
- consult with other teams who are doing a good job of recognition and bake that in to QA
- develop a framework which incorporates Reps as a key stakeholder in the QA contribution model, give local leaders an opportunity to lead
4. Conduct a SWOT analysis [DONE]
- the greatest strength is it's service as a gateway to connect contributors to projects
- the greatest weaknesses are a lack of diversity in topics, a heavy investment in organization, and time/language barriers
- the greatest opportunity exists in working with Reps in locales where passion for Mozilla is high
- the greatest threat is a perception that testdays are irrelevant
Phase 2: Analysis
The purpose of this phase is to analyze the feedback gathered in Phase 1 and collect that into a set of actionable ideas.
| Finding | Proposal |
|---|---|
| Events are too generic and focus on the wrong metric
The goal of these events should not be to get some number of bugs filed against a product or feature but to increase participation in projects that need testers. Critical to this is not just connecting contributors to projects but also ensuring contributors are equipped with the skills and knowledge to make valuable contributions. |
Create diverse, relevant curricula of skills
|
| Participation is restricted to those with privilege
For the most part events tend to only be accessible to people who understand English or are available 9-5 on Friday. Mozilla's strength is in a global community; a restrictive participation model places us in a position of fundamental weakness. We need to evolve these events into something that is far more inclusive and builds on this strength. |
Work with Reps to enable
|
| Etherpads are too fragile and confusing
A couple years ago we switched from using wiki pages for test plans to using Etherpads to make it easier for people to participate and be recognized. The current finding is that the opposite has been true. We've lost significant data because of people inadvertently wiping or overwriting important information (some of this is caused by over-zealous translation add-ons). We've also lost potential contributors because they opted to chat in the often ignored chat widget instead of on IRC. |
Retire the use of Etherpads
|
| We are blind to our community
Apart from a few core members, we know very little about the people in our community (who they are, they're knowledge and capabilities, why they choose to stay or leave, and how they discovered us). Speaking from the side of a participant, they have very little understanding of what we do and why we do it -- a truth that is shared by people internal to Mozilla but external to QA. The result is a feeling unappreciation and irrelevance. |
Develop a shared understanding
|
| We have a lot of disparate communication channels
Over the years our communication channels have grown. From IRC and mailing lists to social networks and forums; there's a lot of options. Unfortunately we don't always know where and how to broadcast our message. We need to get better at this and get our message to the people. A part of this is going to be better understanding of the channels we use, which ones are effective and which ones are not. However we also need to be researching new channels. As the amount of communication increases it will become more important to streamline and automate as much as possible. |
Streamline our communication methods
|
Phase 3: Resolving Dependencies
- WORK IN PROGRESS
The following dependencies, i.e. the proposals that came out of Phase 2, are required to make the Testdays program successful. Not all of these dependencies are hard requirements to roll out the new framework. However, the more we dependencies we resolve the more successful the program will be. These will form the basis of the roadmap for the remainder of 2015 and beyond.
Create diverse, relevant curricula of skills
- skills needed for traditional projects involved in testdays
- skills needed for non-traditional projects in need of contributors
- curricula should include translated documentation and videos
- curricula should include One & Done tasks for self-paced learning
- curricula should connect participant with a mentor
- events should be no longer than a couple hours
- look to models like Fedora Classroom
Work with Reps to enable
- self-paced learning in non-English languages
- events in non-English languages
- events in a physical location
- events any day/time of the week
- participation following an event
- broader evangelism of events
Retire the use of Etherpads
- simplify event organization with a template/checklist
- go back to using wiki pages for test plans
- make sure relevant documentation is on MDN
- integrate video-based communication as a core component
- develop a system to track/reward participants
Develop a shared understanding
- ensure instructions have clear steps, requirements, and reasoning
- develop profiles of people participating in events
- frequently evangelize a brand that is connected to all Mozillians
- frequently solicit feedback from participants
- organize meet-ups and brown-bags once in a while
- develop a strategy to improve participant recognition
Streamline our communication methods
- document effective use of all our communications channels
- identify opportunities to automate event evangelism
- develop a way for people to subscribe to future events
- investigate channels that are under-utilized or unknown
Phase 4: Develop a Roadmap
This roadmap incorporates the dependencies identified in Phase 3 with a goal of implementing a draft framework.
Create central repository of diverse, relevant curricula of skills
1. Create a catalog of skills needed for traditional testdays
- list the projects we've traditionally seen at testdays in the past
- list the skills that are relevant to each of these projects/topics
2. Create a catalog of skills need for non-traditional testdays
- list the projects that have not traditionally been involved with testdays
- list the skills necessary to contribute to testing for those projects
3. For each of the skills identified make sure we have
- an article documenting the skill, any prerequisite knowledge, what will be learned, how it can be applied, and why its important
- a video tutorial designed for self-paced training of the skill
- a One & Done task designed to hone the skill
- a point of contact for asking questions about the relevant material
- a reward and progression plan for when the skill has been "mastered"
4. Run some pilot events, no longer than a couple hours, designed to vet the curriculum
5. Once the requisite material is in place find reps who can translate the material
