Overview

We did a lot of small group brainstorming and problem solving, working on paper (old school!). What follows is my best attempt to type up all the sheets. If I felt that an addition was necessary to help clarify the text, those are in parenthesis, but otherwise I captured exactly what was on the papers. And if I couldn't read something, I put a question mark after the word (sometimes offering several suggestions). Spelling errors are of course mine alone.

These are grouped into the sessions that the sheets came from. Since there were small groups in each session, you'll note that the sessions are repeated.

Cross-Functional Groups

Group 1

Marcia
Johan
Jbonacci
Sydpolk
Juanb
Paul Yang
Geo
Mwargers

Group 2

Jsmith
Kate
Lizzard
Mschifer
Pdehaan
AaronMT
Edward Chen
Mwobensmith

Group 3

Tchung
Zac
Jrgm
Rbillings
Krupa
Edwong
Gaby2300
Kamil

Group 4

Stephend
Drno
Raymond
No-jun
Dylan
Rpappa
Pragmatic
Kthiessen

Group 5

Tracy
Mbrandt
Eric Chang
Kbrosnan
Kairo
Nhirata
Capt. Calliope

Things we Should Change

• Photo
• Not Enough Time Issues - Meetings, Triage, real work, bugzilla, finding, getting answers
  • Possible sulitions Sharing, more people, better docs/sources, indexing, cross-training, more generalization, reasons (having good reasons for why you spend time on something)
• Community - Lowering the bar to entry
  • Good first bug - example
  • One & Done- example
  • Importance
  • Impact,
  • Interest
  • Documentation
  • Inclusive
  • Mentoring & Training programs with services and feedback ala grow Mozilla
• Docs - how to improve our documentation
  • full time community based doc coordinators
  • make full use of MDN
  • doc/learning hub
  • Build team knowledge maps, ReadTheDocs, wiki, etc
• Metrics & Data - we have lots data what's important? what do we do with it?
  • Solutions: What questions need to be answered (vs want)
    • What's the ease/importance/priority of the dat & metrics
    • Requirements gathering - who is your audience

Fx OS Group Priorities

Data & Metrics

• Photo
• monitoring memory over time, see trends
• stability, running hours - load on devices
• better characters of metrics early detection of crashes, OOMs
• organize technical data for analysis
• data on real devices, user in the field
• bug metrics from other branches

Technical Acumen

• expand test automation (on/off device)
• establish technical skills, mentorship
• more testcases to balance the manual ones
• adding supplimental tools to manual testing (i.e. memory logging)
• more facetime with developers, debugging, reproducing
• more technical documentation (i.e. flashing, automation, centralization, location
• working with developer tools, MDN
• strategies more independnce /autoonomy with contract sourcing
• more whitebox testing not all areas have clear user stories
• Knowledge chain? (i.e. API)
• peer-programming, training, tiger-team
• more automation into try-server (ie device testing w/developers)

TOP 3 Photo

• Expand test automation
  • Developer porting, support
  • Adding tests
• Knowledge sharing
  • Documentation, pair-wise/tiger team community training, contractors
• Expanding support tools, memory, logging, flashing, documentation

Community Hypotheses (that we can test)

• Photo
• Office hours online and see if retention of contributors is affected
• Central/unified communication to see impact on retention
• worldwise representation - for ex student ambassadors
• Adding Contributor pathways
• Test visible recognition's impact on contributor retention

Technical Acumen Session

• Photo
• Photo
• Help people realize what we can do: approach with the idea and posture of "How can I help you?" and "What is fragile?"
• Trainings within QA around how tools and automation and technologies work
• Publicize or create a tool for how to get debug info on all platform and products
• Socorro and bugzilla apis (like decoder's auto-verifier) could be really useful tools
• Consistency in quality bug filing when working in your area helping tie to big picture
• Build and apply patches - but those aren't official so understand when to build your own
• Get try access (we all should have it)
• memory tools (reading and creating the reports - knowing what to look for is a learning task)
• ASAN builds
• Leveraging knowledge and bringing it into your area
• creating minimized testcases for for bugs
• ask developers - what kind of tools do you use?

What kinds of things can we do

Photo

• Create a buddy system among us and among us and dev
• QA show and tell bug more interactive
• how to do risk analysis
• replace the QA meeting once a month with a training
• understand why we are testing this thing (what ever it is) and to what degree that testing is needed
• Use things that already exist - code firefox videos, overholt's boot camp
• Videos, wiki pages, with step by step instructions (web maker has great formats for this)
• How to get tools you need -- and, how do you know what tools you need?
• Using tech to separate concerns (like the FMD end to end test)
• Demand testable products
• learn to use 3rd party tools - fx dev tools, wireshark etc

Platform QA Priorities

• Photo
• Photo
• Photo
• Data/Metrics might be a knowledge map - using platform chart?
• Platform is cross product tech
• what is breakdown of automation/manual?
  • Platform should be 100% automation
• other teams have to do full stack as part of their QA
• We need a technical vocabulary
  • allows prioritization of technical leaning
  • pulls together disparate experience
  • chart of what is in the platforrm
  • how do i get started with a platform project

Techncial acumen

• Knowledge of what is included in platform - vocabulary
• platform team is inheretly technically capable

Metrics

• coverage of platform
•  % automations
• utilization of technologies in product
• stability metrics/risk for each area
  • crashes
  • find/fix rate in each area
• investigate all usage metrics -> quality metrics

Community

• training more capabilities
• filtering non-productive/non-<something> charts? RAMP

Things To Change

• Photo
• Photo

Time to explore/try new things

• blockage: manual testing preventing exploration
• recognition of people who make extra effort
• managers pushback to justify time
• individual contributors need to learn time management
• setting aside x% of time to explore - i.e. google's 20/80 rule, freaky fridays

focus metrics on questions

• dev thinks of metrics
• we can learn after the fact?
• disconnect between dev and QA (analyzing the data)
• learn and iterate > readjust
• dashboard - helps visibility
  • does it help just the team or those outside of QA?
  • shared visibility (audience)

lower bar to entry

• get rid of IRC! (hipchat? + forum)
• way for people to ask questions and not get lost (form or feedback)
• beginning Qa questions, ask sumo for help
• community receptionist
• like stack exchange?

knowing who to talk to

• also internal
• like "lower the bar"
• leverage mozillians.org w/roles vs QMO
• add org tree to mozillians

being part of decisions

• mainly outreach to dev and prodcut folks
• two way responsibility ~ dev/qa
• pro-actively interogating dev
• being a domain expert AND show the value you provide (evangelism)
• SOLUTION:
  • regular 1 on 1 with developer manager
  • status updates on QA behalf
  • status update dashboard?

Docs

• each person on team needs to review?
  • major housecleaning of docs?
• single source of truth
• point out outdated docs @ QA meetings
• manager buy in

Things to change

• Photo

lowering bar to entry

• leveling
• enabling first steps
• takeaways from webmaker
• teach teh teacher
• bite size tasks
• community call
  • introductions
  • calls to action
  • tools for teaching the teacher
• Have IRC sheriffs who are pinged when a "newcomer" asks a question - might mitigate "lack of response" problem

Community Hypothesis for Testing

• Photo
• Community calls that are:
  • open
  • accessible
  • consistent
  • compassionate
• will increase community participation by x%
• Weekend testing will bring inn a new type of contributor
  • weekendtesting.com
  • ask for comp day in exchange for running event
  • testing a new product & teaching skills
  • have some of us participate in one of their events first
  • type: professional testers who want to learn something new, different or to test a new product
  • goal: 3-5 people contributing outside of this effort

Services + Web QA Vision And Strategy

• Photo
• Photo
• Photo
• Photo

Technical Growth Areas

• community metrics: tools + workflow
• downstream impact metrics: influence over adoption
• process: too many dashboards!!!!
• robots!
• Docker:
  • streamline test environments
  • selenium?
• better maintain scripts for test coverage?
• Better deployment tests for scale

Vision

Break rules, reassess risk.

Technical Acumen

• Unit tests against public servers
• Streamlining test environments
  • Docker
  • Independently testable services
  • "Web-based IDE for Selenium tests"
• Continuous deployment
  • One-click deployment testing
• Empower contributors to
  • understand APIs
  • run jenkins test
• Requirements Reviews!
• Collaboration/Communication with UX, pincer attack on project lifecycles

Data & Metrics

• Uptime
• adoption
• page response time (#2)
• user happiness
• contributor task analysis
• performance #1
• identifying develpoer blind spots

Community

• independently testable services - packages
• test days
• DIY sync
• crowdsourced baseline data (for science)
• get community to QA test QA break the rules, reveal blind spots, opportunities
• engage community expertise: create "beta programs", "QA Labs"
• localization capacities
• more defined community roles

Community Hypothesis

• Photo
• every person spends a few hours to interact/engage with the community
• redefine success as qa
• dedicated resources to grow qa contributors
• identify projects which are not time-bound for community engagement versus those which are and see which does better
• lots of testdays with local flavor

deeper dive on the time bound idea

• Photo
• time bound projects are not suitable for community
• controls
  • time-bound
  • non-time bound
  • is there pressure in projects with a deadline?
  • no deadline to tasks
• Does a time bound task put under pressure and discourage contributors or will it motivate them more?
• if project owner reports to relman that task is not done because community did not complete it?
  • Will tony lose his job?

Community Hypothesis

• Photo
• if we create a training reward system we will retain contributors
• If we provide a space with more devices we'll attract more potential testers

Unknown Team (FxOS?) Things we should change

• Photo

Time

• Prioritizing tasks/cutting out non-esssential stuff
• Friday afternoon for new tools /fun stuff
  • 10% fun (Tony's team)
• Empower people to do things differently
• Insist on better meetings (< 30 mins)

Metrics

• Better planning /testing based on metrics

Lower bar to participation

• easier set up
• clear docs on what impacts are associated with changing system settings
• VMs/VNC
• community-driven (isolated) projects
• community-driven local/regional testing

Knowing who to talk to

• office hours
• shorter testdays with moderators
• stack exchange for QA

Being part of the decision

• "Testable" is a requirement for anything that ships

Docs

• Less
• Accurate

Unknown? Random Sheet

• Photo
• Test reliability coverage
• Streamlining onboarding + workflows
• Data->Information->Knowledge->Wisdom

Firefox Team Priorities

Technical

• Photo
• P2 - Feature Coverage
• P3 - Working build system:
  • automated tests, bisection, apply patches
• P2 - Deep Dive into feature test coverage
• P1 - knowing where to focus
• P2 More engagement with devs
• P3 Attract technical people & train newcomers
• P3 Learn CSS, JS & underlying web technologies (and specifications)
• P3 Bug lifecycle & process
• P2 - Dev Talks
• P1 Identify areas of Technical Debt
• P3 - Skills to build tools

What kind of data do we need (data/metrics)

• Photo
• automated test results
• api docs, stories
• bug velocity
• bugs w/no tracking
• community

Mschifer's Things to change group

• Photo
• Photo
• Photo

Time

• Know when you are wasting time & do something else
• flex day - make time
• fix/create tools to automate processes
• priority - need way to access priorities (scrum) & complexity
• pairing more to help you get unstuck
• spending too much time chasing down information

Knowing who to talk to

• Bugzilla search to src? who commited
• People who spend time in triage know more abot who to talk to
• Tools to map feature/components to contributors

== Lower bar to entry (free beer)

• Create tutorials/classes to give basic skills (video)
• Define dependencies for tasks & have docs on how to meet them (releated to the above point)
• provide pre-configured VMs
• have a wiki on createing tutorials
• re-evaluate your work to find easy, actionable tasks

Docs

Make docs a priority part of the process

• clean up the docs day
• make a bahit of blogging our activities to inform the rest of the community
• more user friendly:
  • more pictures
• clear steps
• use MDN
• document check points status
  • current processes
  • Todos (storytelling)

Part of decision

• build merit/reputation
  • show you have knowledge of the issue
  • be Proactive
• attend triage & participate
• go to dev meetings & lunches/activities
• pick an area to be an expert in and build ties to that team
• include other sides? spies? specs? in meetings and attend those outside of your normal tasks

Community Hypothesis (Cross-Functional Group 4)

• Photo
• If we make a questionairre (skill matching survey) would that maintain (or create) interest?
• If we pull together pathways, resources, and recognition we can retain more people?
• If we could set up people for success, we could achieve greater coverage of existing projects and perhaps take on more we previously couldn't (Orchestration tools [one and done], bugs ahoy!])