RapidRelease/ImpactAssessment

Preamble

We've made the change to a regular schedule of releases. This is a massive win for our ability to compete. We can deliver new features and fixes to users much more quickly. We can upgrade the web on a continuous basis and respond to changes we see with agility. We can execute better on our mission.

The central benefits of our release schedule aren't the subject of this review. We are keen, though, to ensure we've identified any challenges that this process has introduced. In some cases, we are already well underway on addressing them. In others, we may decide no change is needed at this time. In still others, we may not know the problem exists until someone tells us.

Please help us fully understand the impact of our new process so that we can make smart adjustments where appropriate. If you have concerns you don't see addressed below, please add them, or email and I'll add them.

The owner of this document is Johnathan Nightingale.

Market/Product Impacts

The most important drivers of change in our process should be the impacts our process has on our users and the web.

User Experience

Update Pain

Impact: Firefox updates result in several kinds of distracting interactions that bother our users. These include Firefox internal prompts, ecosystem impacts like add-on compatibility, and OS level interference from things like the Windows UAC security system. Some of these are specific to the major version increments in the new process, some of them have happened all along, even for minor releases. The net effect is a terrible user experience, distraction for our users, and a resistance to apply updates which harms our ability to deliver great software.

Resolution: There are several pieces of work needed to address this impact:

• Eliminating or adjusting our own prompts. This work is largely complete or in progress.
• Add-on compatibility is an area of considerable investment, described below
• Eliminating OS-level interruptions. This work is underway and targeted at early 2012 release. It goes well beyond reducing our level of update annoyance to FF4 levels, in many cases silencing 100% of the interruptions that updates represent (something Firefox has never been capable of previously).

Update Frequency

Impact: We currently ship a release every 6 weeks, barring exceptional circumstance, but it is not clear whether this is the optimum schedule.

• An accelerated schedule (e.g. every 4-5 weeks) would reduce the amount of change that each release represents, likely resulting in better risk management. It would also shorten the time to market for new features. However a shorter cycle would amplify many of the impacts discussed in this document.

• A longer release cycle (e.g. 8-10 weeks) would, by contrast, likely dampen the effect of many of the impacts documented here. Each release would have a larger set of features for users and web developers, albeit with greater associated code and integration risk.

• A significantly longer release cycle (e.g. 3 months) would require us to build an alternate mechanism for delivering security updates, in addition to the costs outlined above. One suggestion has been to syncopate, with security-only releases interwoven with "feature" releases, however these proposals have thus far lacked a clear plan for how testing would accommodate the two streams (particularly since the feature releases would now have twice as much change to test within the same time frame).

• Right now our development, stabilization, and beta testing phases are of equal length. Moving to a shorter or longer cycle may work better for some phases and worse for others. If, f.e. we moved to 10 weeks of dev, would we also need 10 weeks of stabilization and 10 weeks of beta?

Unresolved: This topic requires specific discussion and planning, as the various options have a complex set of costs and benefits.

Channel Confusion

Impact: The current model has four channels: Nightly, Aurora, Beta and Release. This many channels may be a source of confusion for users. This is particularly true for mobile, where Release and Beta are in the android market, but Nightly and Aurora are available only through sideloading/direct download.

• Are our channels providing us the coverage we need?
• Are our channels providing users the right offer to attract the coverage we need?
• Would three channels work as well as four and carry less confusion?
• Are we capable of differentiating and sustaining four channels?

Unresolved: This may not be an issue, or may be an issue only on mobile, but in any case requires a more detailed discussion.

• As a "non-urgent" topic, differentiating and growing these distinct channels has not had much focused investment to date.
• Product Marketing is developing and pushing an Aurora program to strengthen that channel, and work with SUMO to build a support community around Aurora.

Firefox Ecosystem

Add-on Compatibility

Impact: Firefox add-ons check compatibility based on version number, and our new release process changes version numbers frequently, resulting in significant problems with compatibility. Our add-on ecosystem is a central competitive advantage for Mozilla, and something we value highly, as do our users. Concerns about add-on compatibility occur frequently in feedback about upgrades. (It's not just about the version revs, it's that we actually rev XUL "API"s a lot more frequently too.)

Resolution: We are addressing this issue in several ways:

• Automated compatibility checking for AMO hosted add-ons. This is complete, and results in very high compatibility rates from one version of Firefox to the next.
• Third party add-on opt-in. In Firefox 8 we shipped support for informing users when a third party installs add ons and asks for explicit consent. We believe that a significant portion of incompatible, non-AMO add-ons are installed without our users' understanding. Making this choice explicit for users will allow them to disable addons they don't want, removing them from the list of add-ons that will warn about compatibility.
• Changing Firefox's code to treat addons as compatible by default. This work is largely complete, but not yet enabled in shipping Firefox builds and it's not clear what level of compatibility among "dark matter" add-ons that this will get us.

Unresolved:

• We have not yet articulated a clear strategy for add-ons with binary components, who will not be helped by the compatible-by-default work mentioned above.
• We haven't resolved how we're going to ensure the best possible blacklist or mitigation for when the blacklist misses a bad add-on.

Enterprise

Impact: Enterprise practices for qualifying software deployments require significantly longer windows than our release cycle accommodates. As a result, they have expressed significant concern about their ability to continue supporting Firefox. This loses us goodwill, but also means we are failing to serve a significant number of users.

Resolution: Kev Needham is driving the creation of an Extended Support Release version of Firefox, in consultation with our community and the revived Enterprise Working Group. The proposal is near completion, and EWG participants seem supportive.

Unresolved: While the handful of EWG participants seem supportive, we have no real data on whether this will be an effective program -- especially for education and public sectors which are not well represented on the EWG and may have different requirements.

Market Support for Deprecated Fx Versions

Impact: Some site operators, notable Google may adopt a policy of only supporting current and previous major versions of browsers, meaning that Firefox users on relatively recent browser may find themselves shut out of certain web services even though their browser is technically capable of supporting them.

Resolution: Ultimate resolution is to deliver silent updates and then work with the user base to migrate them to that train. This will likely be an iterative process, with a final push after silent updates to Firefox ship. In some respects, there is an opportunity to work with site operators to migrate the Firefox user base to latest versions.

Unresolved: Impact is currently theoretical, and resolution should come in 2012.

Perceptions of the Project

Version Number Inflation

Impact: Some industry followers, and especially web developers and site owners, have expressed concerns about Firefox's major numbering convention. The merits of major version numbering by default, (allowing for the possibility of new features in every release) do not necessarily communicate themselves, and with Chrome and IE both iterating their version numbers more quickly than they have in the past, a sense lingers in some parts that version numbers are being iterated for user-facing "marketing purposes", and that the project is placing the most superficial of concerns above the needs of users and developers.

Resolution: A concerted communication about this topic seems unlikely to generate much more understanding as the topic is a nuanced one. The market will gradually adapt to the Firefox and Chrome schemas, but this will only be resolved once the wisdom of updating version numbers is proven.

Unresolved: Evangelism and online engagement can only do so much; but likely our efforts have not been concentrated on addressing this misconception. This should be addressed over time as new features ship and the differences between versions become more meaningful for developers and less meaningful for users.

Internal/Project Impacts

Second to user impact, but still important, are the challenges our process presents for our own ability to build, ship, and communicate about Firefox.

Localization

Potential Missed Cycles

Impact: While recent feedback shows a slight majority (55%) of localizers prefer the new process, some smaller locales worry that they might miss a cycle and leave users stranded.

Resolution: This concern can be mitigated in some cases by shipping the locale with the new strings untranslated. We have done this already in a few cases where the newly added strings were in low-visibility parts of the product, like the about:permissions prototype.

Per-Release Overhead

Impact: There is a fixed cost associated with spinning up each new release's translation work. This administrative cost is incurred regardless of the number of strings needing translation. This gives localizers the sense that they are burning more time on tiresome administration instead of actual localization.

Unresolved: This issue has not been resolved, though it suggests finding ways to reduce the administrative burden where possible.

Product Management

• Delivering collections of features in a single release is more difficult with shorter cycles than longer cycles. This isn't important for all features, but some collections are bigger than the sum of their parts.
• If all features must ride the trains, that's a minimum of 3 months of the feature being in public builds which may allow competitors to scoop us on features. This isn't important for all features, but could matter for some.

Product Marketing

Focused Communications and Messaging around Feature Collections

Impact: The strength of our outward communications has been greatly challenged by the new release process. This is an issue because Mozilla is much more reliant upon earned media (PR/news) rather than paid media (advertising). So, when things happen, we have to capitalize. As Product notes, it is very difficult to deliver a collection or set of themed features in a single release. This in turn, makes it very hard to articulate product narrative or theme with each update. As a result, we're forced to lean on "laundry list" approach to messaging which greatly dilutes the outward communication's strength and positioning.

Resolution: This concern will be addressed when we move to silent updates. Having silent updates will allow us the flexibility to to wait for a set of features to land and to create messaging around certainthemes instead of having a release define the themes for us. This should enable us to have stronger, more targeted messages that will strengthen positioning in the market. To be clear, we will still make lots of noise when major features land, but this model allows us to have clearer, more concise messaging and to pick and choose what features we talk about. Until silent updates land, we will continue with our current messaging and communications approach.

Building Development Channels

Impact: As noted above, there is a lot of discussion around the development channels, especially the value add of Aurora and Beta. We have an opportunity with release channels to grow a new brand to target influencers. Having a much more precise relationship with early adopters will be enormously beneficial, but requires investment.

• The PMM team is currently working on building the Aurora user base, but part of the issue we're seeing is that users aren't seeing game-changing features land in Aurora (as PMs also mention above), making it hard to distinguish it from the rest of the channels available. We know that many key features are landing soon, but the lack key feature additions in our releases right now is impeding channel differentiation, and in effect, impeding the growth of the Firefox Aurora channel. This will be an issue regardless of how many channels we have, but it should be noted that the growth of this channel will correlate with the impact of the features we land.
• The Beta Channel in our current scheme has very little differentiation because right now it serves as the "more stable Aurora". Although the current focus is on building Aurora, we will see issues when we try to build the Beta channel because of the lack of channel differentiation.

Resolution: Unresolved. As mentioned above, a larger discussion needs to happen around release channels.

Staying Up to Date with Product Changes

Impact: It's very difficult for the PMM team (and other down stream teams) to stay up to date with what features that are being added, removed or held in each release. The Release Tracking Page is useful, but many are not up to date, particularly for Platform features. This has improved overall, but this is still an issue especially as many decisions are made during triage meetings that many are not able to attend. We lack a uniform communications process to make sure the decisions made in Triage are communicated out to downstream teams on a consistent basis. To be clear, this isn't a new struggle--this was also a problem before the move to the new release release, its just been exacerbated by the faster release process, especially if the feature pages are out of date. We also don't expect 100% compliance, but stakeholders should understand the need behind these pages to help (perhaps?) create an expectation of the "normal" feature development process.

Resolution: Unresolved. This issue has not been resolved and will likely need input from PMs, Project Managers and Engineering to find a better process.

Feature Pages are Incomplete

Impact: Many feature pages are out of date or do not include the necessary information to define the feature, use case or explanation of the feature. This is particularly true of platform features that lack explanations that non-techie, down stream teams need to be able to use the feature pages properly.

Resolution: Unresolved. This is something that the Release Tracking page owners, PMM and the platform team will need to discuss.

Engineering

Review bandwidth

Impact: We have preliminary data from Martin Best's work that suggests review backlogs may have been growing disproportionately since the introduction of the new process. It's not immediately clear what is driving this but, if true, it would suggest a need for changes in our development process to avoid languishing in bugzilla instead of shipping, which threatens product quality and agility.

Unresolved: Research is ongoing

Bug Triage

Impact: We have preliminary data from Martin Best's work that shows a growing backlog of untriaged bugs. This might suggest we are missing important bugs due to a lack of resources applied to triage, which threatens product quality.

Unresolved: Research is ongoing

Quality Measurement

Impact: Independent of triage in particular, we are concerned that our current process does not adequately detect breakage during the Nightly->Aurora->Beta migration. We note that so far each rapid release we've shipped has required at least one chemspill (see below), albeit in 2 of 4 cases this was due to third party code change or vulnerabilities that would not have been detectable during testing.

Resolution

• Lucas Adamski is driving an effort to build a set of quality programs which should allow us to track things more closely and detect issues sooner
• Early detection would also be helped by larger testing populations on our Aurora and Beta channels. Growing these populations is a priority for the product marketing organization.

Unresolved: In several cases (e.g. crashes), we do have the metrics now to be able to gauge trends and compare our prior monolithic releases with our new scheduled ones. I'll need to gather this data.

Emergency Response

Impact: Each release under the new model has required a chemspill response at some point during its lifetime. Only half of these were a result of bugs in mozilla code, but it nevertheless remains the case that we have a frequent need to address bugs between scheduled releases. Chemspills are resource intensive and, because they involve deploying new client software for the affected users, can result in significant interruption and annoyance for our users.

Resolution: Christian Legnitto and Dave Townsend have proposed and built support for a new "hot fix" add-on in Firefox to eliminate many situations where we would otherwise require a chemspill. If the chemspill is something that an add-on can address, we can package and deliver code through our daily add-on update system which can address the issue until an actual fix is delivered in a subsequent Firefox release. While hotfixes still require QA and testing and validation, they can be delivered without the full machinery of a release, making us both more agile and less irritating. In most cases, the download, install, and activation of a hotfix would be completely invisible to users.

Contributors

This is the list of people who have contributed, or been invited to contribute, to this process review. If I've forgotten you, please add your name below.

• Johnathan Nightingale - Firefox Engineering
• Christian Legnitto - Release Management
• Asa Dotzler - Desktop Product Management (in progress)
• Erica Jostedt - Product PR (in progress)
• Laura Mesa - Product Marketing
• Chris Hofmann - Localization
• Axel Hecht - Localization (in progress)
• Sheila Mooney - Engineering Project Management (in progress)
• Patrick Finch - Product Marketing
• Kev Needham - Partners (in progress)