Confirmed users
14,525
edits
Releases/Firefox 23/Test Plan: Difference between revisions
Jump to navigation
Jump to search
→Failures
| Line 1,609: | Line 1,609: | ||
** Was not reported until nearly the last Beta | ** Was not reported until nearly the last Beta | ||
** WebRTC testing was not a high priority in our Beta testing | ** WebRTC testing was not a high priority in our Beta testing | ||
*** come up with a better way to track high priority areas that aren't necessarily in the scope of feature testing and the existing Beta checklist | |||
*** work closer with Asa and Developers to get greater visibility, not just in features but high-risk landings | |||
*** add features from current release to regression suite for next release or two | |||
* {{bug|901944}} Rendering glitches on H.264 video only in FF23 on Vista | * {{bug|901944}} Rendering glitches on H.264 video only in FF23 on Vista | ||
** High-risk areas need coverage on a broader range of platforms/hardware | ** High-risk areas need coverage on a broader range of platforms/hardware | ||
** What happened here proves a quite serious communication problem between QA and dev. The QA working on the H264... support feature was told it's only for Windows 7 and 8 and never got further updates. {{bug|847267}} (which caused {{bug|901944}}) wasn't added as a dependency to the tracking bug either {{bug|799318}}. The feature information tracking should get improved. Otherwise we might get left with the only option of spamming devs with emails requesting for updates every week (a situation unpleasant for everyone). | ** What happened here proves a quite serious communication problem between QA and dev. The QA working on the H264... support feature was told it's only for Windows 7 and 8 and never got further updates. {{bug|847267}} (which caused {{bug|901944}}) wasn't added as a dependency to the tracking bug either {{bug|799318}}. The feature information tracking should get improved. Otherwise we might get left with the only option of spamming devs with emails requesting for updates every week (a situation unpleasant for everyone). | ||
** I think we might be getting to a point where visibility of engineering work to QA (or lack thereof) is negatively impacting the quality of the product | ** I think we might be getting to a point where visibility of engineering work to QA (or lack thereof) is negatively impacting the quality of the product | ||
*** status emails are sent prior to uplift but that might miss certain uplifts -- should increase frequency of these emails | |||
*** use need-info request on tracking bugs to get the information we need, and block on it if necessary | |||
* {{bug|904001}} Block rlnx.dll, pmnx.dll, opnx.dll, prnx.dll 1.3.334.9 (Relevant Knowledge 1.0.2) | * {{bug|904001}} Block rlnx.dll, pmnx.dll, opnx.dll, prnx.dll 1.3.334.9 (Relevant Knowledge 1.0.2) | ||
** We were unable to verify this due to old versions of the installer not being available | ** We were unable to verify this due to old versions of the installer not being available | ||
*** Part of a bigger problem, don't have an extensible mechanism to block libraries before we load Firefox | |||
*** Matt to connect Security Assurance team, Brian Bondy with the Stability team | |||
* {{bug|902532}} Spellchecking broken with non-ASCII characters in profile path | * {{bug|902532}} Spellchecking broken with non-ASCII characters in profile path | ||
** Discovered following release | ** Discovered following release | ||
** Can/should this be covered by automation? | ** Can/should this be covered by automation? | ||
*** Not in Mozmill, possibly in Marionette -- carry this discussion forward | |||
** Can/should we be doing more edge-case testing than common-case testing? | ** Can/should we be doing more edge-case testing than common-case testing? | ||
*** look at our coverage and where our blind spots are | |||
*** any automated test involving strings should cover strings with non-ASCII characters | |||
* {{bug|902173}} "[fr] Outils" wrongly translated to "Outils de développement" in menu since FF 23 | * {{bug|902173}} "[fr] Outils" wrongly translated to "Outils de développement" in menu since FF 23 | ||
** Locale-specific issue reported to us in Release, was not in any other branch | ** Locale-specific issue reported to us in Release, was not in any other branch | ||
** Can/should we be engaging with localizers to be involved in RC sign-off? | ** Can/should we be engaging with localizers to be involved in RC sign-off? | ||
*** run an l10n testday for RC | |||
*** reach out to Axel to see if we can get localizer feedback during the RC cycle | |||
* {{bug|902349}} crash in nsStyleSet::FileRules with AMD Radeon HD 6310 | * {{bug|902349}} crash in nsStyleSet::FileRules with AMD Radeon HD 6310 | ||
** We were not able to reproduce this issue despite having hardware previously known to reproduce these types of issues | ** We were not able to reproduce this issue despite having hardware previously known to reproduce these types of issues | ||
*** We are doomed unless we can get engineers engaged in fixing this | |||
*** QA is doing everything we can, Marc to talk to Bob about dev involvement | |||
*** Tracy: Is it the same machine in RelEng farm building the "broken" builds? | |||
** Question whether doing twin RC builds is worth the QA/RelEng resources as compared to just respinning on occasion | ** Question whether doing twin RC builds is worth the QA/RelEng resources as compared to just respinning on occasion | ||
*** QA needs to run full automation on both builds, regardless if we don't ship build2 | *** QA needs to run full automation on both builds, regardless if we don't ship build2 | ||
| Line 1,631: | Line 1,651: | ||
*** After 4 releases (24 weeks) dealing with this, can we do some statistical, risk, and cost-benefit analysis to justify doing this further? | *** After 4 releases (24 weeks) dealing with this, can we do some statistical, risk, and cost-benefit analysis to justify doing this further? | ||
** For further consideration, what's our strategy if we have no more leads and have to just live with this issue? | ** For further consideration, what's our strategy if we have no more leads and have to just live with this issue? | ||
* Features lists for this release were not centralized anywhere, so at least one feature did not get to QA: Network Monitor. | * Features lists for this release were not centralized anywhere, so at least one feature did not get to QA: Network Monitor. | ||
* The beta scope was not adapted to the new 2 betas per week process, which created QA overhead. | * The beta scope was not adapted to the new 2 betas per week process, which created QA overhead. | ||
| Line 1,636: | Line 1,657: | ||
**Unclear tests should be updated. If their owner can't do that, the QA team is willing to take on that. Some of these tests can lead to invalid bugs (e.g. bug 900457). | **Unclear tests should be updated. If their owner can't do that, the QA team is willing to take on that. Some of these tests can lead to invalid bugs (e.g. bug 900457). | ||
**Some of these tests are present in the ESR smoke tests too (e.g. save and open test, play videos test etc - we took the liberty of updating where it was only neccesary to change the links in order to be able to run the tests). | **Some of these tests are present in the ESR smoke tests too (e.g. save and open test, play videos test etc - we took the liberty of updating where it was only neccesary to change the links in order to be able to run the tests). | ||
*** Otilia to get SV to roll up unclear/failed tests into the Release/Beta testing emails | |||
=== Mitigating Strategy === | === Mitigating Strategy === | ||