SWOT Analysis 2014

The following documents a review of strengths, weaknesses, opportunities, and threats of the Testdays program for 2014. It serves as an archival account of how the program will change in 2015.

Meeting Details

• When:
  • Part 1: Friday, December 19, 2014 at 13:00 Pacific
  • Part 2: Monday, December 22, 2014 at 12:00 Pacific
• Where: https://v.mozilla.com/flex.html?roomdirect.html&key=WBuu8ks3P4IZ ('Anthony Hughes' room on Vidyo)
• Notes: https://etherpad.mozilla.org/testdays-swot-analysis-2014

Discussion

Miscellaneous Ideas

• seeking out new avenues of outreach
• trying different activities that draw people in
• find expertise to help us with evangelism (social media for example)
• soliciting known working solutions for building community around program
• engage more with individual "employees" to see what a testday could do for them
• we should do a brown bag to showcase the new testdays and how it's valuable
• we need to define a set of metrics to measure the impact/success so we can make rapid adjustments as needed
• we need to have a feedback mechanism for each testday (eg. survey) if nothing else than to ask people how it made them feel (perhaps integrate with QMO post)
• can we use a snippet to advertise testdays?

What are the strengths of Test Days?

• Good chance to learn lots of new things
• Currently there isn't any strengths for more advanced contributes as most of the verification tasks are pretty basic..This doesn't mean they don't have any value, it's definitely a great way for someone to learn the basics of the work flow but we should definitely start adding some more complex tasks.. This way, there's opportunity to always learn something new.
• Testdays have always been an entryway to get involved with different projects within Mozilla
• have some events focused on mentoring newcomers
• have some events focused on mentoring more technical topics
• exploit more the fact that testdays are an entry point to specific projects
• encourage people who already know these skills to come teach
• use the testday as a gathering/meeting to discuss, establish other activities for people to work on
• have shorter events focused on testing/learning a specific thing - create a framework for this so people involved with different projects within Mozilla can support events

What are the weaknesses of Test Days?

• They are usually weekdays, and often limited to certain time zones.
• Since they cover more time zones it's very difficult to find moderators online to ask questions to or to be taught new abilities
• Hurdles to participation
  • having access to hardware, devices, tools etc to test
  • having a good public calendar to see when events are happening
  • having events only in english
  • Not enough recognition
• have shorter but more focused events but provide more ways to participate outside of the event
• create a blueprint for teams to follow to create their own events
• create better documentation and outreach so people not available for the smaller window can participate
• try to find a way to enable local communities to self organize events
• de-emphasize that an event needs to be online
• engage one on one with key people/groups to design a testday that delivers value for them (what are the artifacts that are valuable to their project) - design the requirements ahead of time and work one on one with those individuals
• find teams that have on-boarding problems due to device/tool access and work together to solve that problem
• reach out to reps to have "localized" events
• set up a google calendar for future events
• be much more proactive with recogniztion (QMO, meetings, badges, tshirts, etc) - look to other teams to see how they're recognizing contribs

What opportunities exist to make Test Days better?

• Make sure there are enough moderators to cover all time zones
• We definitely need more NA contributes to start coming to our test days. It's usually busy during the European hours but as soon as it hits noon or so, the entire channel basically dies. This is why there's a lack of Moderators because most leave as it dies down. Maybe we can implement some type of gamification.. As soon as you have a leader board, people get competitive :) Take a look at Khan Academy.. Everyone loves learning because they get badges, points etc.. We should look into something like that.. Make it more fun and interactive for contributes.
• QA leaderboard for Moztrap, One & Done, Bugzilla, etc
  • reach out to SUMO about their dashboard
  • talk to Clint about getting this on QA's goals
• Shorter more focused events
• We need to get a clearer understanding of why people do not attend (work with CBT)
• Find some allies within the employee base who can commit to engaging with us

What threats have weakened test days?

• In my case, the absence of moderators.
• I think weekend test days is a great idea and we should probably try it a few times to see if users would even turn up.. I honestly don't think it will be to difficult to find a few moderators that would be willing to help out for 1-2 hours on the weekend (I would). The main problem is if users would be willing to show up on weekend. Perhaps we can have a poll on our new QA website and see if contributors would be opened to weekend test days?? I think this is definitely a path we should look into. I remember a lot of people were opened to this idea during our QA work in MTV.
• People are just too busy
• People don't see the value a testday provides to them
• Maybe there simply aren't that many interesting bugs to find. If you're participating but find no problems, did you just waste your time? Could you have a simple extension that helps you gather and show some basic statistics (total test day browser hours of usage, total number of distinct sites visited during testing, my share of those efforts n%) so that a contributor feels the work "counts for" something?

What would make you want to participate in test days?

• Plenty moderators, interesting subjects, (not just the new Beta over and over). It's much more interesting to test new features and the like but active moderators is a must in this case. New Moztrap testcases or new kind of tests (Doing the same Moztrap tests over and over again can be boring)

Agreed with the above comment, as I mentioned earlier.. I think people want to learn new things rather than doing the same boring task over and over. Those are great to introduce people to work flow, processes etc.. but we definitely need more to draw more people to our test days.

• • the real problem is getting people the help they need when they show up
  • employee interest is another problem - Anthony needs to talk to Clint about that
• interesting subjects
  • beta testdays have been used as a basic filler for no topic
  • engaging with project teams to have a short focused testday will help make things more interesting
  • focusing on new features is more interesting than something broadly based on Beta

(Why do we have manual Moztrap tests? Can't they be automated to free up valuable volunteers for other tasks?)

• Moztrap tests are still valuable as an initiating task
• We should provide people with other skillsets tasks that are challenging to them and not just push them to Moztrap tests
• A good quarterly goal would be to have tasks divided into small, medium, large
  • Beginner intermediate advanced
  • Think about the year-long vision, break that down into quarterly goals which break down into tasks to achieve those goals
  • Agreed with the above comment, as I mentioned earlier.. I think people want to learn new things rather than doing the same boring task over and over. Those are great to introduce people to work flow, processes etc.. but we definitely need more to draw more people to our test days.
• Easy entryway
• Opportunity to meet with developers and those involved with the project

What is the return on doing Test Day events?

• If this means organizing them, getting to know lots of new people who may start contributing, teaching them new abilities,
• When I was a contributor, I personally went to several test days so I could meet the team behind Mozilla and learn as much as I can about the browser space when it came to QA. I think most people come so they can learn and add more into their tool set. (and we should provide this) Some people help just because they want to feel like they're doing something to improve the open web. Maybe we can send people stickers that contribute on a regular basis? Maybe a shirt for the person that verified or completed the most tasks for the month? I was never more motivated and happy when Matt sent me a Mozilla shirt!!

How would Test Days best serve you, your team, and your future work?

• this question should be answered every time we approach a team or individual about organizing a testday
• define a roadmap of how we want to make testdays more vital
• run quarterly pilots to be project focused and regional (in line with 2015 for CBT)
  • talk with David Tenser and Christler Khoeler to align with their goals
• work with product groups and pilot

What is the 2015 vision, how do we get there? try to "bucket" different kinds of tasks

• localization, regional issues
• recognition
• low hanging fruit
• tools
• draw inspiration from the Webmaker roadmap

Conclusions

• On strenghts, testdays are one of the best ways to get involved with projects within Mozilla and to teach skills necessary to be successful in the projects. We need to refocus these events to make them a vital tool for bringing new contributors to various projects.
• On weaknesses, testdays are too broad, do not serve people with developed skill-sets, and are sometimes inaccessible to people due to barriers like language, time, and access to technology. They require heavy investment to organize and provide little return on that investment. We also do a very poor job of recognize volunteers who support and participate in testdays. We need to develop ideas that resolve specific barriers to entry, and to have events that focus on provide more advanced skills (not just the fundamentals) that are relevant to specific projects.
• On opportunities, testdays can be more vital to Mozilla, to connect projects to people to skills to regions, to improve recognition across Mozilla, and to reduce barriers to getting involved with Mozilla. We need to identify areas of Mozilla which have community building challenges and collaborate on solving those challenges.
• On threats, testdays are threatened by a lack of participation (particularly mentors and moderators) which results in people not being able to get help. The general concensus is that people are too busy and testdays do not present enough value. We need to work closer with specific people to bring volunteers into their projects and thereby create value.
• On participation, the general feeling is we need to create more activity over a condensed time frame, to improve participation, to have a greater variety/depth of topics, to bring new people into projects but to also enable people to grow within those projects.
• On value, testdays present a unique opportunity to draw a lot of people to a single project, to provide people the skills necessary to contribute, and to connect volunteers to real Mozillians.
• On service, testdays can best serve the future of any given project by developing events in close collaboration with that project, to determine the valuable outcomes and tailor events to those needs. We need to develop a long-term vision, a roadmap, and tasks that can be subdivided into achievable accomplishments at two week intervals.

Next Steps

1. Define a vision for 2015
2. Develop a list of action items to achieve that vision
3. Categorize and prioritize the list of action items into tasks which are achievable in two weeks
4. Divide tasks into quarters and define quarterly goals based on tasks