Firefox/Go Faster/Process - Revision history

Ckprice: Move

2016-10-28T18:26:43Z

Move

← Older revision		Revision as of 18:26, 28 October 2016
Line 1:		Line 1:
	~~This is the process used to release system add-ons. This Wiki page was created based on [https://docs.google.com/document/d/1x27I7hAmWDWiqk3o3YC3fklhE3N59bdgHCQHF5p_lkU/edit this Google Document].~~		MOVED TO: https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process

	~~= New System add-on planning and development phase =~~

	~~== Initial Notification==~~

	Once a decision has been made to implement a new system add-on, dev owner will send an “Intent to Implement” email to '''release-drivers@mozilla.org''' and '''gofaster@mozilla.org''' and '''firefox-dev@mozilla.org''' mailing list.

	* Reference (bug, repository, Wiki):
	* Overview of the add-on:
	* Timeline (including development and testing):

	~~The responsible EPM will add this as a card to~~ https://~~trello.com/b/moJCpVCD/go-faster-system-add-on-pipeline and tracked in the Go Faster checkins.~~

	~~== Implementation Complete ==~~
	~~Devs are encouraged to give QA a heads-up a few weeks before the expected system add-on implementation complete date. This will help QA plan testing resources and staffing needs.~~

	~~== Testing system add-on ==~~
	~~System add-on code and uplifts will go through testing via Nightly, Aurora and Beta channels. QA team will work with devs to create and review a test plan and test schedule.~~

	~~= Uplift process for GoFaster fixes =~~

	~~== Beta Uplift criteria - Best practices ==~~

	~~After Beta 6, we only uplift fixes for:~~
	* Recent critical regressions, blocking bugs
	* top crashers affecting browser stability and
	* sec-high/crit vulnerabilities

	~~Keeping in mind the general Beta uplift criteria, we recommend that GoFaster changes be planned such that:~~
	* Bulk of new system add-on work needs to land in m-b before Beta6
	* Before uplifting to Beta, the patches need to stabilize on Nightly, Aurora.
	* All ground work needed to get the system add-on working should be code complete and uplifted to moz-beta before Beta6. Only fit-and-finish type fixes will be uplifted beyond Beta6 in Beta cycle
	* A few minor uplifts can land post Beta6 if deemed critical.
	* <Need to add a suggested timeline for localization related changes that need uplifting>

	~~= Process to Push updates to release channel =~~

	~~System add-ons updates that are pushed live on release channel should follow the steps below:~~

	~~== Verification on pre-release channel ==~~

	~~Update must be tested on Beta/Try (by dev) and validated (by QA) before requesting push to release.~~

	~~== Notification to release-drivers@mozilla~~.~~org and gofaster@~~mozilla.org ==

	~~“Intent to ship” tracking bug should link to all the bugs that were fixed as part of the system add-on update. This will be needed eventually for sec advisory pushes that happen through GoFaster.~~

	~~Intent to ship email (see draft below) to <release-drivers> and <gofaster> ML and a corresponding bug to track. Plan to send this email 4-5 days before update-go-live date.~~
	* uplift request made by dev (mozilla-approval-release = “?”)
	* uplift approval made by RelMan (mozilla-approval-release = “+”)
	* relnote-firefox tracking flag set by dev with suggested wording for relnote
	* steps to push the change live

	~~== Multiple system add-on updates ==~~

	~~For multiple system add-on updates that need to be pushed in parallel, we need to combine test efforts so these occur in parallel to save QA and ship time.~~
	* We need one “intent-to-ship” tracking bug for each system add-on update.

	~~== Uplift request ==~~

	~~Before landing the code, dev owner will request mozilla-release uplift.~~
	~~Uplift request will be reviewed by RelMan~~

	~~== Updating release notes to announce system add-on update ==~~
	~~If approved, RelMan will create a corresponding release note with tag “UPDATE [datestamp-of-update-go-live]” in the main release notes and set to public after the update is pushed live.~~

	~~== Staged rollout ==~~
	* Starting with staged rollout to 10% users, dev owner will push the change by landing the code in moz-release branch and balrog configuration change.
	* If 24-48 hours of data indicates no regressions, the system add-on update will be pushed out to 100% end-users.
	* Comms will publish blogs only after system add-on update is unthrottled and at 100%.

	~~= “Intent to Ship” email =~~
	* Bug Id: <bug #>
	* System add-on: <add-on name> <add-on version>
	* Committed to Moz-Release: <Yes/No, Commit Id if Yes>
	* Verification: <Indicate fix was verified or new feature was signed-off>
	* Other areas of impact: <describe any impact to stability/~~security~~/~~perf~~/~~mem-usage>~~
	* Proposed date to go live: <go-live-date>
	* Relnote suggested wording: <Wording suggested by dev in bug when nom’ing for relnote addition, also add a link to the code change/commit>

	~~= System add-on updates on pre-release vs. release channel =~~
	It is important to mention that updates on pre-release channel involve uplifting changes to mozilla-central/mozilla-aurora/mozilla-beta branch and the subsequent build contains the updated system add-on and no changes are pushed via balrog server.

	If there are changes to the system add-on update process (update server like Balrog), we will test the update mechanism on Beta channel before pushing to release. Otherwise we can go straight to release channel and push directly after QA signs off on the system add-on updates.

	~~For updates to release channel, the patch is uplifted to mozilla-release but the update is pushed to release channel via balrog.~~

	~~= Scheduling of system add-on updates on release channel =~~

	GoFaster team discussed the idea of releasing system add-on updates on pre-fixed days of the week. Instead of doing pre-fixed days, we will list out system add-on update blackout dates - these are the days when we should not plan to ship an update.

	~~== Blackout dates ==~~
	* Mozilla-wide work weeks (June and December)
	* Holidays: July 4th, December 25th, December 31st
	* Week of release go-live

	~~= History of release version and system add-on updates pushed=~~
	~~Ian made a comment about maintaining a history of all releases and add-on versions pushed to the respective release. Balrog and release notes can be used to extrapolate this info.~~

	~~We may need some tooling support to make this happen. TBD~~

	~~= Open Issues =~~
	* This Wiki seems to focus mostly on in-tree addons with possible uplift. But not necessarily addons that are specifically developed for old versions
	* When will QA begin testing? When the add-on work is complete or when the add-on implementation code enters Aurora cycle?
	* When do we add telemetry probes? This should be included in test plan discussion. Is there a way to do this for system add-ons?
	* System Add-on A/~~B Testing~~
	* Laura brought this up as a topic of discussion for a future meeting

	~~= Ideas for release process v2 =~~
	~~Consider setting a release cadence for system add-ons if we start seeing a lot of pushes during the release cycle.~~

	~~Some suggestions:~~
	~~Add-on updates (that are not critical) are pushed on Tuesday and Thursday of the week only. This builds more predictability in dev, QA and release efforts.~~
	~~Updates that fix critical issues can occur on a as-need basis.~~

Ckprice: /* Notification to release-drivers@ and gofaster@ */ Update email

2016-10-14T00:49:42Z

Notification to release-drivers@ and gofaster@: Update email

← Older revision		Revision as of 00:49, 14 October 2016
Line 43:		Line 43:
	Update must be tested on Beta/Try (by dev) and validated (by QA) before requesting push to release.		Update must be tested on Beta/Try (by dev) and validated (by QA) before requesting push to release.

	== Notification to release-drivers@ and gofaster@ ==		== Notification to release-drivers@mozilla.org and gofaster@mozilla.org ==

	“Intent to ship” tracking bug should link to all the bugs that were fixed as part of the system add-on update. This will be needed eventually for sec advisory pushes that happen through GoFaster.		“Intent to ship” tracking bug should link to all the bugs that were fixed as part of the system add-on update. This will be needed eventually for sec advisory pushes that happen through GoFaster.

Ckprice: More updates

2016-09-22T23:23:08Z

More updates

← Older revision		Revision as of 23:23, 22 September 2016
Line 9:		Line 9:
	* Reference (bug, repository, Wiki):		* Reference (bug, repository, Wiki):
	* Overview of the add-on:		* Overview of the add-on:
	* Timeline:		* Timeline (including development and testing):

	~~This~~ will ~~then appear~~ as a card in https://trello.com/b/moJCpVCD/go-faster-system-add-on-pipeline and tracked in the Go Faster checkins.		The responsible EPM will add this as a card to https://trello.com/b/moJCpVCD/go-faster-system-add-on-pipeline and tracked in the Go Faster checkins.

	~~== Schedule ==~~

	A first draft of the timeline that describes when the system add-on will go through planning, development and testing phase will be included in the “Intent to Implement” email. GoFaster EPM will help in establishing this timeline.

	== Implementation Complete ==		== Implementation Complete ==

Ckprice: /* Initial Notification */ Updates to email

2016-09-22T23:18:28Z

Initial Notification: Updates to email

← Older revision		Revision as of 23:18, 22 September 2016
Line 5:		Line 5:
	== Initial Notification==		== Initial Notification==

	Once a decision has been made to implement a new system add-on, dev owner will send an “Intent to Implement” email to '''release-drivers@mozilla.org''' and '''gofaster@mozilla.org''' and '''firefox-dev@mozilla.org''' mailing list. This ~~email is similar to~~ the ~~email notification that is sent out for new feature development work~~.		Once a decision has been made to implement a new system add-on, dev owner will send an “Intent to Implement” email to '''release-drivers@mozilla.org''' and '''gofaster@mozilla.org''' and '''firefox-dev@mozilla.org''' mailing list.

			* Reference (bug, repository, Wiki):
			* Overview of the add-on:
			* Timeline:

			This will then appear as a card in https://trello.com/b/moJCpVCD/go-faster-system-add-on-pipeline and tracked in the Go Faster checkins.

	== Schedule ==		== Schedule ==

Ckprice: /* Open Issues */ Added Spohl's question.

2016-08-10T16:50:58Z

Open Issues: Added Spohl's question.

← Older revision		Revision as of 16:50, 10 August 2016
Line 100:		Line 100:

	= Open Issues =		= Open Issues =
			* This Wiki seems to focus mostly on in-tree addons with possible uplift. But not necessarily addons that are specifically developed for old versions
	* When will QA begin testing? When the add-on work is complete or when the add-on implementation code enters Aurora cycle?		* When will QA begin testing? When the add-on work is complete or when the add-on implementation code enters Aurora cycle?
	* When do we add telemetry probes? This should be included in test plan discussion. Is there a way to do this for system add-ons?		* When do we add telemetry probes? This should be included in test plan discussion. Is there a way to do this for system add-ons?

Ckprice: /* Notification to release-drivers@ and gofaster@ */ Adding in detailed instructions.

2016-08-10T16:46:01Z

Notification to release-drivers@ and gofaster@: Adding in detailed instructions.

← Older revision		Revision as of 16:46, 10 August 2016
Line 45:		Line 45:
	“Intent to ship” tracking bug should link to all the bugs that were fixed as part of the system add-on update. This will be needed eventually for sec advisory pushes that happen through GoFaster.		“Intent to ship” tracking bug should link to all the bugs that were fixed as part of the system add-on update. This will be needed eventually for sec advisory pushes that happen through GoFaster.

	Intent to ship email (see draft below) to <release-drivers> ML and a corresponding bug to track. Plan to send this email 4-5 days before update-go-live date.		Intent to ship email (see draft below) to <release-drivers> and <gofaster> ML and a corresponding bug to track. Plan to send this email 4-5 days before update-go-live date.
	* uplift request made by dev (mozilla-approval-release = “?”)		* uplift request made by dev (mozilla-approval-release = “?”)
	* uplift approval made by RelMan (mozilla-approval-release = “+”)		* uplift approval made by RelMan (mozilla-approval-release = “+”)

Ckprice: /* Notification to release-drivers */ include gofaster@

2016-08-09T17:12:11Z

Notification to release-drivers: include gofaster@

← Older revision		Revision as of 17:12, 9 August 2016
Line 41:		Line 41:
	Update must be tested on Beta/Try (by dev) and validated (by QA) before requesting push to release.		Update must be tested on Beta/Try (by dev) and validated (by QA) before requesting push to release.

	== Notification to release-drivers ==		== Notification to release-drivers@ and gofaster@ ==

	“Intent to ship” tracking bug should link to all the bugs that were fixed as part of the system add-on update. This will be needed eventually for sec advisory pushes that happen through GoFaster.		“Intent to ship” tracking bug should link to all the bugs that were fixed as part of the system add-on update. This will be needed eventually for sec advisory pushes that happen through GoFaster.
Line 49:		Line 49:
	* uplift approval made by RelMan (mozilla-approval-release = “+”)		* uplift approval made by RelMan (mozilla-approval-release = “+”)
	* relnote-firefox tracking flag set by dev with suggested wording for relnote		* relnote-firefox tracking flag set by dev with suggested wording for relnote
	* steps to push the change live		* steps to push the change live

	== Multiple system add-on updates ==		== Multiple system add-on updates ==

Ckprice: Create this page.

2016-07-19T17:51:51Z

Create this page.

New page

This is the process used to release system add-ons. This Wiki page was created based on [https://docs.google.com/document/d/1x27I7hAmWDWiqk3o3YC3fklhE3N59bdgHCQHF5p_lkU/edit this Google Document].

= New System add-on planning and development phase =

== Initial Notification==

Once a decision has been made to implement a new system add-on, dev owner will send an “Intent to Implement” email to '''release-drivers@mozilla.org''' and '''gofaster@mozilla.org''' and '''firefox-dev@mozilla.org''' mailing list. This email is similar to the email notification that is sent out for new feature development work.

== Schedule ==

A first draft of the timeline that describes when the system add-on will go through planning, development and testing phase will be included in the “Intent to Implement” email. GoFaster EPM will help in establishing this timeline.

== Implementation Complete ==
Devs are encouraged to give QA a heads-up a few weeks before the expected system add-on implementation complete date. This will help QA plan testing resources and staffing needs.

== Testing system add-on ==
System add-on code and uplifts will go through testing via Nightly, Aurora and Beta channels. QA team will work with devs to create and review a test plan and test schedule.

= Uplift process for GoFaster fixes =

== Beta Uplift criteria - Best practices ==

After Beta 6, we only uplift fixes for:
* Recent critical regressions, blocking bugs
* top crashers affecting browser stability and
* sec-high/crit vulnerabilities

Keeping in mind the general Beta uplift criteria, we recommend that GoFaster changes be planned such that:
* Bulk of new system add-on work needs to land in m-b before Beta6
* Before uplifting to Beta, the patches need to stabilize on Nightly, Aurora.
* All ground work needed to get the system add-on working should be code complete and uplifted to moz-beta before Beta6. Only fit-and-finish type fixes will be uplifted beyond Beta6 in Beta cycle
* A few minor uplifts can land post Beta6 if deemed critical.
* <Need to add a suggested timeline for localization related changes that need uplifting>

= Process to Push updates to release channel =

System add-ons updates that are pushed live on release channel should follow the steps below:

== Verification on pre-release channel ==

Update must be tested on Beta/Try (by dev) and validated (by QA) before requesting push to release.

== Notification to release-drivers ==

“Intent to ship” tracking bug should link to all the bugs that were fixed as part of the system add-on update. This will be needed eventually for sec advisory pushes that happen through GoFaster.

Intent to ship email (see draft below) to <release-drivers> ML and a corresponding bug to track. Plan to send this email 4-5 days before update-go-live date.
* uplift request made by dev (mozilla-approval-release = “?”)
* uplift approval made by RelMan (mozilla-approval-release = “+”)
* relnote-firefox tracking flag set by dev with suggested wording for relnote
* steps to push the change live

== Multiple system add-on updates ==

For multiple system add-on updates that need to be pushed in parallel, we need to combine test efforts so these occur in parallel to save QA and ship time.
* We need one “intent-to-ship” tracking bug for each system add-on update.

== Uplift request ==

Before landing the code, dev owner will request mozilla-release uplift.
Uplift request will be reviewed by RelMan

== Updating release notes to announce system add-on update ==
If approved, RelMan will create a corresponding release note with tag “UPDATE [datestamp-of-update-go-live]” in the main release notes and set to public after the update is pushed live.

== Staged rollout ==
* Starting with staged rollout to 10% users, dev owner will push the change by landing the code in moz-release branch and balrog configuration change.
* If 24-48 hours of data indicates no regressions, the system add-on update will be pushed out to 100% end-users.
* Comms will publish blogs only after system add-on update is unthrottled and at 100%.

= “Intent to Ship” email =
* Bug Id: <bug #>
* System add-on: <add-on name> <add-on version>
* Committed to Moz-Release: <Yes/No, Commit Id if Yes>
* Verification: <Indicate fix was verified or new feature was signed-off>
* Other areas of impact: <describe any impact to stability/security/perf/mem-usage>
* Proposed date to go live: <go-live-date>
* Relnote suggested wording: <Wording suggested by dev in bug when nom’ing for relnote addition, also add a link to the code change/commit>

= System add-on updates on pre-release vs. release channel =
It is important to mention that updates on pre-release channel involve uplifting changes to mozilla-central/mozilla-aurora/mozilla-beta branch and the subsequent build contains the updated system add-on and no changes are pushed via balrog server.

If there are changes to the system add-on update process (update server like Balrog), we will test the update mechanism on Beta channel before pushing to release. Otherwise we can go straight to release channel and push directly after QA signs off on the system add-on updates.

For updates to release channel, the patch is uplifted to mozilla-release but the update is pushed to release channel via balrog.

= Scheduling of system add-on updates on release channel =

GoFaster team discussed the idea of releasing system add-on updates on pre-fixed days of the week. Instead of doing pre-fixed days, we will list out system add-on update blackout dates - these are the days when we should *not* plan to ship an update.

== Blackout dates ==
* Mozilla-wide work weeks (June and December)
* Holidays: July 4th, December 25th, December 31st
* Week of release go-live

= History of release version and system add-on updates pushed=
Ian made a comment about maintaining a history of all releases and add-on versions pushed to the respective release. Balrog and release notes can be used to extrapolate this info.

We may need some tooling support to make this happen. TBD

= Open Issues =
* When will QA begin testing? When the add-on work is complete or when the add-on implementation code enters Aurora cycle?
* When do we add telemetry probes? This should be included in test plan discussion. Is there a way to do this for system add-ons?
* System Add-on A/B Testing
* Laura brought this up as a topic of discussion for a future meeting

= Ideas for release process v2 =
Consider setting a release cadence for system add-ons if we start seeing a lot of pushes during the release cycle.

Some suggestions:
Add-on updates (that are not critical) are pushed on Tuesday and Thursday of the week only. This builds more predictability in dev, QA and release efforts.
Updates that fix critical issues can occur on a as-need basis.