Fennec/NativeUI/BetaReadinessReview: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(100 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== [Fennec Native] Beta Readiness Review ==
== [Fennec Native] Beta Readiness Review ==


== Approval/Disapproval Needed by: May 14, 2012 ==  
== May 14, 2012 ==  


Decision Makers: Damon Sicore, Jay Sullivan, Chris Beard Gary Kovacs-?, Brenden Eich
=== Beta Release Plan ===
==== Friday- May 11 ====
<strike> [Kevin] analyze the Beta Sign-Off test matrix for an other couple hours so he is 150% sure </strike> <br>
<strike> [Aki] 13 beta 2 XUL build is kicked off for tablets </strike> <br>
<strike> [QA] Once build is available QA verifies 13 beta 2 XUL for tablets. No new regressions </strike> <br>
<strike> [Erin, Alex] Draft of release notes available for review </strike> <br>
<strike> [Erin] Status and timeline released </strike> <br>


=== Blocking Bug Metrics === <== This assumes we don't make any other changes besides beta blockers
==== Monday - May 14 ====  
Find/Fix rate
AM<br>
Blocking Bugs
√ [QA] [https://wiki.mozilla.org/QA/Fennec/Milestones/13.0/Beta/Beta_2/Build_2 Complete 13 beta 2 XUL for tablet verification and sign-off]
Bounce-Back Rate
√ [Alex] [https://wiki.mozilla.org/Releases/Fennec_Native_14_Beta/Final_Signoffs Formal in-person beta release sign-off confirming all x-functional teams have tasks complete]
[All] Submit (optional) blog posts for review to communications@mozilla.com<br>


=== Deferred Bug Metrics ===
PM<br>
List the most worrisome bugs that are being deferred.  List critical deferred bugs that may have Support impact, or which should be considered for the next beta.  Explain why each bug isn't being or can't be fixed.
√ [Alex] Final release notes on staging<br>
√ [Jaclyn] All marketing web site changes on staging<br>
√ [Michelle] All support materials complete<br>


=== Features Summary ===  
==== Tuesday- May 15 ====
Summarize new features that ship with this release
AM<br>
[Alex] Go LiveGoogle Play and  Mobile Beta Website (target 9AM Pacific)
 
=== Weekly Beta Release Schedule ===
 
[05/18] Beta 2<br>
[05/25] Beta 3<br>
[06/01] Beta 4<br>
[06/08] Beta 5<br>
[06/15] Beta 6<br>
[06/22] Beta 7<br>
[06/29] GA, LAST DAY OF Q2
 
*Go-to build every Tuesday, at the latest on Wed afternoons <br>
*Policy: Priority is frequency. However, we will block on serious issues reported by customers Google Play & regressions on the tree <br>
 
=== Marketing Distribution Plan ===  
 
==== Websites being updated/localized/launched on May 15th ====
*[https://www.mozilla.org/mobile/beta/ hero image update]
*[https://play.google.com/store/apps/details?id=org.mozilla.firefox&hl=en Google Play Store content update]
*FAQ for versions and l10n plan
List text (general location is fine) that is being updated and localized)<br>
* Bug 755411 - "About Firefox Beta" Release Notes link is 404<br>
* Bug 756908 - Link to FAQ on About Firefox Beta page points to a missing page<br>
 
==== Other Promos ====
*Facebook ads for messaging testing
*Beta newsletter
*Beta channel Snippets
*Core Mozillians communication
*Reddit IamA campaign


=== Test Coverage ===  
=== PR Intro Plan ===  
Summarize the test coverage status for each functional area.
*5/15 Introduction date <br>
Please note % level of completion of these items (the goal is assumed to be 100%).
*"Future of Firefox" blog post <br>
Note the number of days required to satisfy the goal of 100% completion. If the goal is not 100% then state the goal.
*Friendly reporters have been given a head's up <br>
*FAQ for inbound press questions <br>


=== Reliability ===  
=== Support Plan ===  
List the reliability criteria defined for the project.
*[https://www-dev.allizom.org/en-US/mobile/14.0beta/releasenotes/ Release notes]<br>
State results of reliability testing (measured against the criteria).
*[https://wiki.mozilla.org/Mobile/Platforms/Android#System_Requirements Minimum System Requirements & Device Compatibility]
*https://www.mozilla.org/en-US/mobile/platforms/ Updated Platforms page targeted for a beta very soon]
*en-us support docs complete for Google Play launch; l10n ready
*[http://support.mozilla.org/en-US/mobile#os=android&browser=m14 New Firefox 14 for Mobile Support Site; 60 new articles and new tutorials]
*firefoxforandroid@mozilla.com: one-time push notification about Flash
*army of awesome will monitor social sites and direct problem solving on sumo;
*Keep feedback flowing into Google Play for best support; monitoring throughout the day
*Daily reports for the first week or so, depending upon traffic:
-Positives<br>
-Negatives <br>
-Rating change <br>
-Interesting comments/trending <br>


=== Performance ===  
=== Issues and Risks ===  
List the performance criteria defined for the project.
*en-us only build for beta 1<br>
State results of performance testing (measured against the criteria).
*Web site compatibility effort is ongoing<br>
*Crash rate (although trending in the direction)<br>
*GFX work is ongoing
*Add-On support for beta: {{Bug|744555}}, need add-on developer adoption for compatibility with new architecture


=== [Lukas] Mobile Test Drivers & Usability Testing ===  
=== Deferred Bugs Scheduled for Beta 2 ===
#Number of testers
*[https://bugzilla.mozilla.org/buglist.cgi?quicksearch=717096%2C725389%2C736123%2C748531%2C752426%2C752444%2C752688%2C753534%2C753845%2C754276%2C725389%2C736123%2C739648%2C748531%2C750846%2C751479%2C752052%2C752444%2C752688%2C753534%2C753845%2C754276%2C738676%2C746860%2C747231%2C747857%2C751262%2C752492%2C752514;list_id=3083671 We will continue to approve until the cut-off for go-to build]
#Number of bugs reported
*[https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocking-fennec1.0%3AbetaN%2B;list_id=3083812 tracking issues we want fixed early in the beta cycle with a specific flag
#Validation (Area of focus and results)
*[https://www-dev.allizom.org/en-US/mobile/14.0beta/releasenotes Other notable bugs in Known Issues]
#Activity (Builds seeded and results)


=== [AaronMT, Lawrence] Web Compatibility ===  
=== Features Summary ===  
#Where we are today
*Performance
#Where we want to be when we ship
*click-2-play Flash support
#If goals can't be obtained when we ship in June, provide high level for k9o
*Readability
*Refined UI, better tab management,smoother in-content UI


=== [Erin, Alex] Release Plan ===  
=== Test Coverage ===  
#l10 details <== Erin
#Test cases based on each functional area 100% complete
#Google Play Details<br>
#This includes verification of fixed blockers across a variety of devices, smoke testing, usability, and exploratory testing
#Minimum System Requirements<br> <== Erin
#Installation, update, and migration testing 100% complete
#Google Play pre-testing 100% complete


=== [Erin, Eng, QA] Schedule ===  
=== Reliability ===  
#Weekly betas
#[https://crash-stats.mozilla.com/products/FennecAndroid/versions/14.0a2?duration=3 As of May 14th, we are at 22 crashes per 100 ADU]
#Go-to build dates
#Criteria initially defined was no more than 10 crashes per 100 ADU
#Google Play release dates
#We have not met that goal 100%, however we are trending downward and are continuing to systematically fix top crashes and reproducible crashes
#We have flexibility until about June 5th


=== [Jaclyn] Marketing Distribution Plan ===  
=== Performance ===
• List websites being updated/localized/launched for this beta release<br>
====Goals====
*Fast start up time
*UI is up and running and ready for user interaction quickly
*Responsive browsing: page load, panning & zooming performance
*Better Memory Usage
====Performance State====
*[http://people.mozilla.org/~johnath/fnWe think we are within acceptable beta quality see state of Fennec Perf Section for more details]
*Latest numbers highlighted in weekly status


=== [Erica, Barry] PR & Marketing Intro Plan ===  
=== l10n ===
• [EJ]Introduction date, means by which the product will be introduced (press release), notable events, any blog requirements from Dev/engineering
*Multi-locale builds not supported in beta 1 {{bug|753488}}
• [Barry] Any brand marketing activities/website updates planned
*Multi-locale builds have been turned on and are being qualified
*Spacing for multi-locale languages are too small in portrait mode {{bug|753880}}
*Committed to adding this support as early on in the beta cycle as possible


=== [mluna, Erin, Alex] Support Plan ===  
=== Mobile Test Drivers ===  
• Release notes [alex] https://www.mozilla.org/en-US/mobile/13.0beta/releasenotes/ <br>
*Number of testers: 500
• [mluna] SUMO <br>  <== mluna, I believe you and I are meeting about this tomorrow. :)
*Number of bugs filed to date 144 (this not include bugs filed outside of the MTD form which auto-magically tags them for tracking).
*Areas of focus: Web compatibility, panning and zooming, flash,
*Builds released: 2 specific aurora builds with gfx changes we wanted extra validation on. Otherwise, the updater provides a new build every day.
*We are *keeping* MTD testers on Aurora


=== Issues and Risks ===  
=== Web Compatibility ===  
• State any outstanding issues and/or risks.
*Investigation is ongoing (crowd sourcing test tool launched 20120508), actively working with select partners to fix site issues.
*We are working on top sites. Goals are aligned around k9o. Coordinating with evangelism as needed top priority issues.
*High level goals for k9o are captured in {{bug|747123}}. Are targeting active involvement with top apps/sites and mitigation strategies for Alexa top sites in specific locales.
*Directed testing email was sent to Nightly testers on 05/14 to emphasize web compatibility testing and provided them with tips on reporting issues.

Latest revision as of 18:49, 23 May 2012

[Fennec Native] Beta Readiness Review

May 14, 2012

Beta Release Plan

Friday- May 11

[Kevin] analyze the Beta Sign-Off test matrix for an other couple hours so he is 150% sure
[Aki] 13 beta 2 XUL build is kicked off for tablets
[QA] Once build is available QA verifies 13 beta 2 XUL for tablets. No new regressions
[Erin, Alex] Draft of release notes available for review
[Erin] Status and timeline released

Monday - May 14

AM
√ [QA] Complete 13 beta 2 XUL for tablet verification and sign-off √ [Alex] Formal in-person beta release sign-off confirming all x-functional teams have tasks complete [All] Submit (optional) blog posts for review to communications@mozilla.com

PM
√ [Alex] Final release notes on staging
√ [Jaclyn] All marketing web site changes on staging
√ [Michelle] All support materials complete

Tuesday- May 15

AM
[Alex] Go LiveGoogle Play and Mobile Beta Website (target 9AM Pacific)

Weekly Beta Release Schedule

[05/18] Beta 2
[05/25] Beta 3
[06/01] Beta 4
[06/08] Beta 5
[06/15] Beta 6
[06/22] Beta 7
[06/29] GA, LAST DAY OF Q2

  • Go-to build every Tuesday, at the latest on Wed afternoons
  • Policy: Priority is frequency. However, we will block on serious issues reported by customers Google Play & regressions on the tree

Marketing Distribution Plan

Websites being updated/localized/launched on May 15th

List text (general location is fine) that is being updated and localized)

  • Bug 755411 - "About Firefox Beta" Release Notes link is 404
  • Bug 756908 - Link to FAQ on About Firefox Beta page points to a missing page

Other Promos

  • Facebook ads for messaging testing
  • Beta newsletter
  • Beta channel Snippets
  • Core Mozillians communication
  • Reddit IamA campaign

PR Intro Plan

  • 5/15 Introduction date
  • "Future of Firefox" blog post
  • Friendly reporters have been given a head's up
  • FAQ for inbound press questions

Support Plan

-Positives
-Negatives
-Rating change
-Interesting comments/trending

Issues and Risks

  • en-us only build for beta 1
  • Web site compatibility effort is ongoing
  • Crash rate (although trending in the direction)
  • GFX work is ongoing
  • Add-On support for beta: bug 744555, need add-on developer adoption for compatibility with new architecture

Deferred Bugs Scheduled for Beta 2

Features Summary

  • Performance
  • click-2-play Flash support
  • Readability
  • Refined UI, better tab management,smoother in-content UI

Test Coverage

  1. Test cases based on each functional area 100% complete
  2. This includes verification of fixed blockers across a variety of devices, smoke testing, usability, and exploratory testing
  3. Installation, update, and migration testing 100% complete
  4. Google Play pre-testing 100% complete

Reliability

  1. As of May 14th, we are at 22 crashes per 100 ADU
  2. Criteria initially defined was no more than 10 crashes per 100 ADU
  3. We have not met that goal 100%, however we are trending downward and are continuing to systematically fix top crashes and reproducible crashes

Performance

Goals

  • Fast start up time
  • UI is up and running and ready for user interaction quickly
  • Responsive browsing: page load, panning & zooming performance
  • Better Memory Usage

Performance State

l10n

  • Multi-locale builds not supported in beta 1 bug 753488
  • Multi-locale builds have been turned on and are being qualified
  • Spacing for multi-locale languages are too small in portrait mode bug 753880
  • Committed to adding this support as early on in the beta cycle as possible

Mobile Test Drivers

  • Number of testers: 500
  • Number of bugs filed to date 144 (this not include bugs filed outside of the MTD form which auto-magically tags them for tracking).
  • Areas of focus: Web compatibility, panning and zooming, flash,
  • Builds released: 2 specific aurora builds with gfx changes we wanted extra validation on. Otherwise, the updater provides a new build every day.
  • We are *keeping* MTD testers on Aurora

Web Compatibility

  • Investigation is ongoing (crowd sourcing test tool launched 20120508), actively working with select partners to fix site issues.
  • We are working on top sites. Goals are aligned around k9o. Coordinating with evangelism as needed top priority issues.
  • High level goals for k9o are captured in bug 747123. Are targeting active involvement with top apps/sites and mitigation strategies for Alexa top sites in specific locales.
  • Directed testing email was sent to Nightly testers on 05/14 to emphasize web compatibility testing and provided them with tips on reporting issues.
Retrieved from "https://wiki.mozilla.org/index.php?title=Fennec/NativeUI/BetaReadinessReview&oldid=433875"