QA/FennecESR

From MozillaWiki
< QA
Jump to: navigation, search
Last updated: 2019/05/15

Overview

Purpose

The purpose of this wiki is to serve as a general test plan for testing Fennec ESR Nightly and Beta builds.

Scope

  • Builds should look and act like regular Nightly/Beta builds, with a few exceptions.

QA: *Fennec QA (slack: mobilesv)

Overall Status

  • Done

Testing Summary

Scope of testing

  • As part of the work to move Fennec to ESR, we need to verify that Nightly builds coming from a Beta-level tree function as expected, as this will be the first time we've ever done such a thing.

Environments

Testing will be performed on the following devices:

  • Phones
    • Samsung Galaxy S8 (Android 9)
    • Sony Xperia Z5 (Android 7.0.0)
    • Google Pixel 3XL (Android 9)
  • Tablets
    • Nexus 9 (Android 7.1.1)
    • x86 - Xiaomi Mi Pad 2 (Android 5.1)
    • Samsung Galaxy Tab S3 (Android 8)
    • Xiaomi Mi Pad 2 (Android 5.1)
    • Lenovo Yoga Tablet 2 (Android 4.4.2)

Acceptance Criteria

  • All tests should pass.

Testing details

Test Cases

Nightly and Beta:

Testing days

Date: 2019-05-10

Tracking bugs

  • Bugs to track the feature
Full Query
ID Priority Summary Status Resolution Whiteboard
1505538 -- Release x86_64 Fennec Nightly in the Google Play Store VERIFIED FIXED [geckoview:fenix:p2]
1518056 P1 Can't load any pages on Android Q pre-release builds RESOLVED DUPLICATE [geckoview:fenix:p2]
1520261 -- Let ARM64 Fennec ride the trains to Beta RESOLVED FIXED [arm64:m3]
1545204 P3 Make Fennec tests run on ESR 68 branch NEW
1546320 P1 Create mozconfigs for Fennec beta builds RESOLVED FIXED
1546322 -- Modify merge scripts to not modify Fennec mozconfigs anymore NEW
1548552 -- Stop shipping Fennec nightly starting 69 RESOLVED FIXED
1548655 -- Figure out Fennec version numbering as different channels ride the trains to ESR RESOLVED FIXED
1548659 P2 Move Fennec tests on mozilla-central to Tier 2 and reduced frequency after the Gecko 69 version bump NEW
1548864 -- Build and ship Fennec Nightly off mozilla-beta NEW
1550719 -- Cannot update official Beta/Nightly builds with Fennec-ESR builds NEW
1551442 -- [ESR builds][Tablet] Make default browser option not displayed on top of the Settings Menu RESOLVED INVALID
1552251 -- Clean up Android chunking rules RESOLVED FIXED

13 Total; 5 Open (38.46%); 7 Resolved (53.85%); 1 Verified (7.69%);


Signoff

Criteria

  • All Test cases should pass.
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/QA).

Results

Nightly testing

Date 10-05-2019

  • Fennec-ESR - Armv7 - Nightly 68.0a1 on Google Pixel 3XL (Android 9)
  • Fennec-ESR - AArch64 - Nightly 68.0a1 on Samsung Galaxy Tab S3 (Android 8.0)
  • Fennec-ESR - x86 - Nightly 68.0a1 on Lenovo Yoga Tablet 2 (Android 4.4.2)
  • Fennec-ESR - x86-64 - Nightly 68.0a1 on Xiaomi Mi Pad 2 (Android 5.1)

Beta testing

Date 10-05-2019

  • Fennec-ESR - Armv7 - Beta 68.0b1 on Sony Xperia Z5 (Android 7.0.0)
  • Fennec-ESR - AArch64 - Beta 68.0b1 on Samsung Galaxy S8 (Android 9)
  • Fennec-ESR - x86 - Beta 68.0b1 on Lenovo Yoga Tablet 2 (Android 4.4.2)
  • Fennec-ESR - x86-64 - Beta 68.0b1 on Xiaomi MiPad 2 (Android 5.1)

Issues

  • New:
    • bug 1550719 - Cannot update official Beta/Nightly builds with Fennec-ESR builds
    • bug 1551442 - [ESR builds][Tablet] Make default browser option not displayed on top of the Settings Menu


Example Bug: **{{Bug|1305969}} - Fennec sometimes get stuck in a state where there is a perma-bar of white on the bottom of the screen