QA/Desktop Firefox/Work Weeks/2014 Planning
Contents
Firefox Desktop 2014 Planning Work Week
The following summarizes the notes of the Firefox Desktop 2014 planning work week in Paris, January 13th to 17th, 2014. The goal of this work week was to outline the goals and direction Firefox Desktop would be taking in 2014. The work week included representatives from development, design, quality assurance, business development, user experience, and user research.
TL;DR
2014 will be a time to refine a more tightly integrated and more agile process for all teams aligned to the Firefox Desktop product team. We will all strive to deliver Australis, Firefox Accounts, Instant Translation, First-run sponsored tiles, Plug-in whitelisting, better testing infrastructure for experiments, and a new work process.
Day 1
Day 1 focused on a retrospective on 2013 and planning of priorities for 2014. The first order of business was to set the tone for the week and to outline the high-level goals for 2014.
High-level Goals
The first goal is to increase the average number of hours Firefox is used per user by 10%. This will be achieved by improving how Firefox performs for certain high-impact secondary browser use cases, increased investment in word-of-mouth evangelism, and aggressive outreach in emerging markets. User studies conducted in 2013 revealed over half of Firefox users are using multiple browsers and that emerging markets are showing extremely high potential for growth via word-of-mouth.
The second goal is to increase monthly revenue by $10M incremental to Google funding. It's no secret that a large portion of Mozilla's financials are supported by Google. We will continue to partner with Google while finding new ways to diversify our revenue. This will be achieved by developing new in-product promotional channels that add value to our users, and by implementing an adjacent-to-the-browser user case with monetization potential.
The third goal is to sign-on 25 million new users to Firefox accounts. We've seen that our competitors are already having some success with tying in user accounts and Firefox presents a largely untapped opportunity to promote and leverage other products and markets. This goal will be achieved by integrating Mozilla services which add value to users who use multiple Firefox touchpoints, by improving discovery of these services within Firefox, and by leveraging tent-pole promotional windows to increase user interest in an account.
Metrics
The second talk was from Metrics to talk about some interesting measurements of late and their methodology. The core framework of metrics is to measure acquisition activation, retention, revenue, and referral of our products and services. Speaking to acquisition, in the last six months we've seen 300M FHR profiles activated (170M in the last month) with a median profile age of 10.5 months. Speaking to activation, we've measured 21 billion hours spent online by our users (2 billion "active" hours) resulting in an average of 1.1 hours per day per user. Speaking to retention we've seen ~82% of our users remain active after two weeks with the highest retention rates in Poland and Japan, and the worst retention rates in the US. To support the 10% usage goal, the Metrics team has devised a couple of different indexes to measure usage.
The first index is the performance index. This is an aggregate of startup time, time from first-paint to main, time from session restored to first-paint, and the proportion of sessions that end in an abort. The result is a score from -5 to +5 with higher values correlating to greater usage.
The second index is the usage index. This is an aggregate of mean total session time, proportion of session length that is active, the number of sessions in one day, the proportion of active days to potentially active days, the number of days which pass before users update to the latest version, and the mean changes in pages visited. The numbers from recent releases distinctly show regional holidays (ex. there's a considerable dip in the index for en-US builds in November). The index also shows that Poland is our most loyal user base with much less pronounced "dips" and much faster recovery from these dips.
A third index based on measure how changes in Firefox affect search behaviours is currently in development.
Looking more at FHR revealed a few other interesting metrics. It appears that ~20% of new profiles have add-ons installed by the user, 30% of which continued to use Firefox after installing an add-on, and that creating/deleting bookmarks increased the likelihood of continued browser usage by 7.5%. This points to a need to encourage early usage and involvement with browser features. In other words, if users feel engaged in our product early on they are more likely to remain Firefox users.
Sustainability
The third talk was about financial sustainability. Several ideas were put forward to the steering committee in terms of providing opportunity to generate revenue, ancillary to existing initiatives. These ideas were analyzed based on the amount of time it would take to deliver, whether they truly put the user first, and how much revenue they would potentially generate. Many of the ideas centered on themes of advertising and sponsored content within the browser, particularly in browser whitespace (ie. empty tiles in the New Tab page). Some of the ideas centered on transactional themes like partner based notifications based on browser usage. In the end some ideas were considered to have mutual benefit to our users and our sustainability while some were rejected outright.
Secondary Browser Use Cases
The final talk of the day was about secondary browser use cases. As markets go, we have the best retention rates in the US, Western Europe, and Japan while our growth markets are Mid-east Asia, Africa, and South-east Asia. In terms of users, we currently have about 350 million users with 145 million identified as "active". Looking at secondary use cases it was found that over 50% of Firefox users also use Internet Explorer and/or Chrome, 38% appeared to favour Chrome while 32% appeared to favour IE. However, very few users who primarily use the other browsers consider Firefox as their secondary browser (most IE users use Chrome secondarily, most Chrome users use IE secondarily). This signals a need for us to drastically improve our user story for secondary use cases.
Irrespective of their primary browser, users often switched to another browser mostly for better for search, video, or productivity tasks, more performant for certain websites, for startup, or for certain tasks, better features like add-ons, automatic translation, and search integration, or using another browser allowed them to separate their browsing habits (ie. work vs personal, private vs public, etc)
In general, Firefox is perceived to be the second fastest browser with our best perception being in India. The tasks Firefox users most commonly switch to another browser are for search, productivity, videos, social, file sharing, reading, gaming, streaming audio, and photo sharing. The features Firefox users most commonly switch to another browser for are add-ons, automatic translation, search, customization, data sync, developer tools, and form fill/password management. Interestingly, many people use another browser as a way to physically separate their browsing behaviours (ex. work vs personal, public vs private).
In the end this information tells us that we need to improve Firefox's performance perception, support features like translation services and simultaneous multi-profile, improve optimizations for top-sites like search, video, and social, and improve engagement around technical evangelism, product positioning, marketing, and distribution.
Day 2
Day 2 focused on process. The Firefox Desktop team workshopped the idea of moving development to a 2-week sprint model, similar to what the Metro development team has been using. Looking back on 2013 it was revealed that there were many pitfalls. We didn't have a focused plan and schedule, there was much confusion about prioritization of work, many people were unsure if what they were working on was really valuable to achieving our goals, some work lingered and was left incomplete, and information was not communicated well outside of the team. To solve these issues in 2014 it is proposed that development will move to a two week sprint model complete with a prioritized backlog of bugs, frequent team planning meetings, frequent goals measurement, and frequent demo sessions of work completed.
To begin, a prioritized backlog will exist as a list of work that is considered to be important. During the planning meeting at the beginning of the sprint, highest priority items will be identified with well defined acceptance and completion criteria. In other words, what it means for a feature to be "ready for release". An example backlog can be seen here. This backlog will be groomed by the managers, a process that will be completed before the planning meeting on the first Tuesday of each iteration, at which point the iteration itself is planned. The purpose of this planning meeting will be to kick off the new iteration, review the previous iteration, explain the top-priority work, and allow teams to select their work. Additionally, the planning meeting which occurs closest to branch migration day will have time set aside to review features approved and denied uplift.
The next major component of this model will be feature breakdowns. Anything that is identified as being too much work for one iteration will be broken down into smaller, achievable work. Each of these pieces of work will be assigned to an individual with a corresponding bug report in Bugzilla. Any work that is broken out and deprioritized will be put in the backlog and considered for a future iteration. The goal is to have features which can be individually forecasted, developed, tested, and released. An example work breakdown can be seen in bug 924860.
A critical component of this process will be accurate estimation of the work to be completed. Since it's easier to estimate in relative terms, points will be used to identify work relative to other work in the list. The points themselves will be used to indicate the "size" of the work, not how much time it will take to complete the work. Ultimately, the more points a team completes in an iteration, the greater the team's velocity. Points are awarded using the fibonacci sequence so that the value of the work completed grows non-linearly. The points will be broken down as such:
- Each work item gets placed into one of three buckets (easy, medium, difficult)
- Any work in the easy bucket is awarded 1, 2, or 3 points (using 2 points as the benchmark)
- Any work in the medium bucket is awarded 5 or 8 points
- Any work in the difficult bucket is awarded 13 points and is flagged for a work breakdown
Status meetings will occur once mid-way through the iteration. This meeting will be used to discuss development status of all the work previously selected for the iteration and to adjust expectations of what's achievable. The status meeting closest to the Nightly->Aurora migration will also include public demos by the UX designers and developers assigned to the features being uplifted.
The plan will be to start in the Firefox 30 cycle with three two-week iterations within the six week development cycle. This will allow for two weeks of scoped feature development and one week of stability/quality fixes. In general each two-week iteration will proceed as follows:
- Day 1: planning and work selection for the iteration
- Day 2 - 5: development of selected work
- Day 6: managers groom the backlog
- Day 7: meet to discuss iteration status
- Day 8 - 12: development of selected work
- Day 13: managers groom the backlog, meeting to demonstrate work completion
- Day 14: iteration wraps up and new iteration begins
In terms of QA it is thought that we will keep up with verify fixes and doing manual regression testing no more than one iteration behind. Nothing that hasn't been verified (or qa-'d) will be allowed to ride the trains or uplift to Aurora/Beta. It should be emphasized that it will likely take some time to adjust to this model and iterate several times before we get it right. One obvious risk is that Bugzilla does not support indicating bug priority relative to other bugs, as a result some tooling will need to be but in place. In the mean time this will require some micro-management by leads. It should also be emphasized that the Platform team lacks the structure to support such a model so they will not be adopting this model for the time being.
Day 3
Day 3 started with a brainstorming session where we were all invited to post ideas to drive the goals outlined earlier in the week (ie. revenue, usage, and accounts). Madhava has the post-it notes and can be contacted if you want to know more. Following this session we had a presentation from Chad about how to pitch ideas to the development team.
Pitching an Idea
The first talk in the morning was about pitching ideas to the development team. Chad outlined that all ideas should have clarity, focus, accountability, and transparency. Ideas should focus on our goals and strategy, come with crisp user stories and use cases, contain a value proposition, provide supporting materials to help understand the core concept, provide preliminary estimations of resourcing, contain a core hypothesis, and a defined testing plan. An example he gave was for translation in Firefox. The hypthesis was that users leave Firefox when they want to view content in another language, if we provided translation the hours of Firefox usage would increase. The associated cost of implementation would include access to instant translation APIs since building in-house APIs would cost significant personnel resources. Ultimately the scope was reduced to shipping a build with a bundled add-on to pre-release channels for 3 release cycles to a couple of target markets and to measure adoption through FHR. Under the new Firefox Desktop development model, the managers will meet every three weeks to hear pitches from individuals.
Q1 2014 Goals
The second talk was about defining the 2014 Goals. The following lists the Firefox Desktop goals for 2014:
- Australis in Beta (Usage)
- Accounts authenticated Sync in Beta (Accounts)
- Instant Translation pilot launched (Usage)
- First run sponsored tiles on mozilla-central (Revenue)
- Plugin whitelist launched (Usage)
- Spec for A/B testing infrastructure to evaluate new features/experiments (Meta)
- New work processes up and running (Meta)
Metro is not "Firefox Desktop" goal as it's a separate product team, although it's scope to ship in Q1 2014. It will be treated as a separate product, just like Firefox for Android. The decision to leave off any community-oriented goal was because the management team believes success in community supports delivery of these goals.
Technical Talks
The afternoon was more technical in nature, talking about some of the initiatives which are taking place to improve Firefox's perception in the world. The first talk was about how e10s was solving some of our technical performance problems but could not really solve the perception problem Firefox has. Some initiatives that the async team will be working on is chrome workers, xpcshell workers, asynchronous tests, off-main-thread downloader, off-main-thread I/O, asychronous primitives, asychronous debugging, asychronous shutdown, faster worker communication, data journaling, and improved IO speeds.
The next presentation was about designing in four dimensions, or about how design principles can lead to completely different perceptions. One example given was that when Australis landed we received a lot of feedback that Firefox was faster, in spite of not landing a single speed improvement. The result of redesigning the UI and UX has led to Firefox "feeling" faster. More examples of perceptive design can be found on Philipp Sackl's github page.
The second to last presentation was about user research findings in South-east Asia, focusing on Indonesia and Thailand. The purpose of the study was to understand how people in these regions were using Firefox. Some interesting findings was that we should probably be optimizing more for poor connections, increasing our growth through word of mouth, providing physical distribution channels, simplifying the entry points to search, include a predictive search feature, educate people about relevant browsing features, providing more integrated translation, educate users more about security and privacy, and inform people more about Mozilla's mission of web literacy and community.
The day ended with a demo of Mike Deboer's Chromeless2 API for standalone, distributable applications with all Firefox platform features available, including a command-line tool, access to OS features, and NodeJS Package Manager modules.
Day 4
The morning of the fourth day was reserved for developers and designers to work on Australis bugs. In the afternoon we had a retrospective on 2013 and the overall feel of the work week. Common themes throughout this talk were that people feel clearer about 2014 than they did for 2013, the interactive exercises were really useful to demonstrating the changes in process, the mix of three presentation days to two hacking days felt good, and having representatives from all teams was extremely productive.
Day 5
The fifth and final day was left for hacking, as some of the people had to travel home.
Questions
- 2014 Goals:
- How can QA support these goals or is this just an outcome of us doing our jobs?
- Should QA be re-aligned to the engineering teams instead of spread across Desktop, Metro, Platform, etc? Should we have a single "Firefox QA" org with resources aligned to the various "products" and a separate "Platform QA" org to focus on core products (Graphics, JS, NSS etc)?
- Metrics:
- Are there FHR/Telemetry probes QA could be using to help guide future testing strategies?
- Should QA be looking at regional browser usage to help guide future testing strategies?
- Process:
- How has this process worked for Metro QA? benefits & concerns?
- Should QA desktop structure similarly (ie. 2-week QA iterations)?
- How can we QA the last iteration before uplift to Aurora? Should that last iteration be the "stability" iteration?
