QA/Fennec: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
 
(47 intermediate revisions by 6 users not shown)
Line 1: Line 1:
Home of those who want to Propel Fennec Quality!
= Overview =
The team focuses on testing current and upcoming releases of Firefox on Android. Some of our responsibilities include qualifying builds prior to a maintenance or milestone release. Some of the responsibilities include, but are not limited to:


== Milestones ==
* Verifying bugs using a triage strategy
* 7.0: [https://wiki.mozilla.org/QA/Fennec/Milestones/7.0 Testplan]
* Running basic functional tests and smoke tests - mostly manual; but automated in the near future
* 6.0: [https://wiki.mozilla.org/QA/Fennec/Milestones/6.0 Testplan]
* Running our manual test suites on localized builds
* <strike>5.0: [https://wiki.mozilla.org/QA/Fennec/Milestones/5.0 Testplan]</strike>
* Making sure web content is viewable, acceptable and ready for consumption in Firefox for Android
* <strike>2.0: [https://wiki.mozilla.org/QA/Fennec/Milestones/2.0 Testplan]</strike>
* <strike>1.1: [https://wiki.mozilla.org/QA/Fennec/Milestones/1.1 Testplan]</strike>
* <strike>1.0: [https://wiki.mozilla.org/QA/Fennec1.0/TestPlan Testplan]</strike>


== Planning ==
In addition to qualifying builds prior to a release, we perform ongoing tasks on the development branches to ensure no major problems get uplifted as we merge changes to the next branch. For example, on a daily basis we:
* [https://wiki.mozilla.org/QA/Fennec/Planning/2011 2011]
* [https://wiki.mozilla.org/QA/Fennec/Planning/2010 2010]


== What We Do ==
* Confirm new unconfirmed bugs
* Verify bugs on development branches like Aurora and Beta
* Engage community and new volunteers in helping improve Firefox through test day events
* Interact with developers to help them test the features they implement
* Perform exploratory testing on new features while on the Aurora branch
* Write test cases in [http://moztrap.mozilla.org Moztrap] for new features
* Monitor [http://crash-stats.mozilla.com/ crash-stats] to assess stability
* File and track new crasher bugs as they appear in crash-stats
* Manage web compatibility testing through community involvement or crowdsourcing companies that help in testing different Android device configurations and top-site testing


All of which help assess the quality of individual new features as well as the Firefox version as a whole.


=== Testing Strategy ===
= Team Details =
The team is lead by both Kevin Brosnan and Aaron Train who coordinate testing of their respective areas among several MoCo employees and community volunteers. Traditionally new features are tested by several MoCo QA employees. Anyone is welcome and encouraged to participate.


Manual:
* Lead - [mailto:kbrosnan@mozilla.com Kevin Brosnan (:kbrosnan)]
* [[QA/Fennec/Waverly|Daily Smoketests]] on the latest nightly build on the current development branch and trunk
* Lead - [mailto:atrain@mozilla.com Aaron Train (:aaronmt)]
    1. Verifications of overnight resolved bugs on the current development branch
    2. Run BFT(s)/Focused testing around areas that are affected by resolved bugs
* Weekly triage of feedback received via the Fennec Reporter. URLs to the current lists can be found here: [http://reporter.mozilla.org/app/query/?report_description=&report_useragent=&report_gecko=&report_language=&report_platform=&report_oscpu=&report_product=Fennec%2F1.1a2pre&report_file_date_start=&report_file_date_end=YYYY-MM-DD&show=25&host_hostname=&report_problem_type=-1&report_behind_login=-1&&submit_query=Search 1.1a2pre] and [http://reporter.mozilla.org/app/query/?report_description=&report_useragent=&report_gecko=&report_language=&report_platform=&report_oscpu=&report_product=Fennec%2F1.0.0&report_file_date_start=&report_file_date_end=YYYY-MM-DD&show=25&host_hostname=&report_problem_type=-1&report_behind_login=-1&submit_query=Search 1.0.0]
* Weekly triage of filed bugs and those that are nominated and in a New state
* Before string freeze, update [https://litmus.mozilla.org/run_tests.cgi?test_run_id=48 the String Guide] in the latest Localizer Test Run with new features added for that release
* Bi-Quarterly Testdays


Automation:
== Team Members and Assignments ==
  * [https://wiki.mozilla.org/Mobile/Fennec_Automation Unittests on Fennec]
{| class="fullwidth-table"
    1. [https://wiki.mozilla.org/Mobile/Fennec_Mochitest Mochitests on Fennec]
| style="background:#EFEFEF; width:30%" | '''Name'''
    2. [https://wiki.mozilla.org/Mobile/Fennec_Reftests Reftests on Fennec]
| style="background:#EFEFEF; width:30%" | '''Contact'''
  * For more information, take a look at the [https://wiki.mozilla.org/Mobile/Fennec_TestDev TestDev Wiki for Mobile]
| style="background:#EFEFEF; width:10%" | '''Availability'''
  | style="background:#EFEFEF; width:30%" | '''Project Assignments'''
|-
|Aaron Train
|[mailto:atrain@mozilla.com atrain@mozilla.com]
|MoCo Employee (full time)
|Lead - New features testing coordination.
|-
|Kevin Brosnan
|[mailto:kbrosnan@mozilla.com kbrosnan@mozilla.com]
  |MoCo Employee (full time)
|Lead - New features testing coordination.
|-
|}


Unique Areas of Testing:
=QA Current Release Test Plans=
  * [http://wiki.mozilla.org/QA/Fennec/Daily_Browsing Test Guide on Daily Browsing]
{| class="fullwidth-table"
  * [http://wiki.mozilla.org/QA/Fennec/Webpages Website Compatibility]
| style="background:#EFEFEF; width:25%" | '''Project Test Plan'''
  * [http://wiki.mozilla.org/QA/Fennec/Device_Integration Device Integration Checklist]
| style="background:#EFEFEF; width:25%" | '''Description'''
  * [http://wiki.mozilla.org/QA/Fennec/OS_Integration OS Integration Checklist]
| style="background:#EFEFEF; width:25%" | '''Scope'''
  * [http://wiki.mozilla.org/QA/Fennec/Updates Updates Testing]
| style="background:#EFEFEF; width:25%" | '''Timeframe'''
|- valign="top"
  |[https://wiki.mozilla.org/QA/Fennec/31 Firefox for Android 31]
| QA activities revolving around Firefox Aurora in the time frame for Firefox 31 final release.
| [https://wiki.mozilla.org/Mobile/Roadmap#Firefox_31_.28Beta.29 Features in Firefox for Android 31]
| [https://wiki.mozilla.org/Releases/#Upcoming_Releases Upcoming Releases]
  |- valign="top"
|[https://wiki.mozilla.org/QA/Fennec/32 Firefox for Android 32]
  | QA activities revolving around Firefox Nightly in the time frame for Firefox 32 final release.
| [https://wiki.mozilla.org/Mobile/Roadmap#Firefox_32_.28Aurora.29 Features in Firefox for Android 32]
| [https://wiki.mozilla.org/Releases/#Upcoming_Releases Upcoming Releases]
  |- valign="top"
|[https://wiki.mozilla.org/QA/Fennec/33 Firefox for Android 33]
| QA activities revolving around Firefox Nightly in the time frame for Firefox 32 final release.
| [https://wiki.mozilla.org/Mobile/Roadmap#Firefox_33_.28Nightly.29 Features in Firefox for Android 33]
  | [https://wiki.mozilla.org/Releases/#Upcoming_Releases Upcoming Releases]
|- valign="top"
| [https://docs.google.com/spreadsheet/ccc?key=0AhE7m4JB2j6tdGVKQlRHUnFWVEZXNFFJbHo0RVpQMmc Feature Ownership]
| colspan="3" | Testing, Planning, and Bug Triage of new and stable Firefox features used till Firefox for Android 30
|}


=== Release Testing ===
=Community Contribution=
* [https://wiki.mozilla.org/Mobile/Releases/Release_Checklist Mobile Release Checklist]
'''Anyone can participate.''' There are several ways in which '''YOU''' can participate:
* Template for [https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/Template results tracking page]
* File bugs
 
* Triage bugs (confirm existing bugs and assign them to the right buckets)
{| class="standard-table"
* Help test new features
|-
* Write test cases
|class="header"|Release
* Plan new features testing
|class="header"|Smoketest
* Help others who want to get involved.
|class="header"|BFT
|class="header"|Blocker Verifications
|class="header"|L10n
|class="header"|Website
|class="header"|Updates
|-
|Early-Stage Betas
|YES
|NO
|YES
|1 RTL, 1 non-Latin, 1 Multi-L10n Switch
|NO
|YES
|-
|Late-Stage Betas (3+)
|YES
|YES
|YES
|1 RTL, 1 non-Latin, Full Multi-L10n Switch
|NO
|YES
|-
|Major Releases
|YES
|YES
|YES
|2 RTL, 2 non-Latin, Full Multi-L10n Switch
|YES
|YES
|}
 
=== Fennec Crash Reporting ===
 
* [https://wiki.mozilla.org/QA/Fennec/crash_rep_2feb Week of 2/2/2011]
* [https://wiki.mozilla.org/QA/Fennec/crash_rep_01june Week of 6/01/2011]
* [https://wiki.mozilla.org/QA/Fennec/crash_rep_08june Week of 6/08/2011]
* [https://wiki.mozilla.org/QA/Fennec/crash_rep_15june Week of 6/15/2011]
 
== What We Use ==
 
 
=== Tools ===
* [https://addons.mozilla.org/en-US/mobile/addon/87500 Reporter for Fennec]: Extension to allow users to inform Mozilla of problems during their browsing session. For triagers, reports are filed to the following queries, ____ and ____.
* [https://addons.mozilla.org/en-US/mobile/addon/13954 QA Companion]: Currently allows users to painlessly report bugs if they don't have a desktop on hand.
* Litmus [https://litmus.mozilla.org/run_tests.cgi?test_run_id=35 Fennec 1.0 Test Run]
* Litmus [https://litmus.mozilla.org/run_tests.cgi?test_run_id=48 Fennec 1.0 Localizer Test Run]
* [http://graphs.mozilla.org/ Performance Graphs]
 
=== Resources ===
* [http://ted.mielczarek.org/code/mozilla/extensionwiz/ Ted's extension wizard]
* [http://hg.mozilla.org/mobile-browser/ Fennec HG] Repository
* [http://mxr.mozilla.org/mobile-browser/ Fennec MXR Source Code] Web-Based Viewer
* Builds
** [http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-trunk/ Trunk Nightly Builds]
** [http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ Hourly Tinderbox Builds]
** [http://l10n.mozilla.org/dashboard/?tree=fennec10x Fennec L10n] Dashboard
* Helpful Links
** [http://starkravingfinkle.org/blog/2009/11/fennec-nightly-maemo-updates/ HowTo:Maemo]
** [https://wiki.mozilla.org/Mobile/Fennec/Cheat_Codes Fennec Cheat Codes]
* [http://spreadsheets.google.com/pub?key=rdm8c2ZfSDKd5l-dVy4SrnA&output=html Android Hardware Comparison]
 
== Where Do We Need Help ==
 
=== Test Tool Development ===
* Write automated [https://wiki.mozilla.org/Mobile/Fennec_TestDev#Areas_to_focus_on testcases]
* Port [https://wiki.mozilla.org/Mobile/Fennec_Automation Automation] Frameworks to WinCE
* Find, Fix, Verify [https://wiki.mozilla.org/Mobile/Fennec_TestDev#Other_TestDev_related_todo.27s UnitTest] Failures
 
=== Execution ===
* Triage [https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced;bug_status=UNCONFIRMED;product=Fennec Unconfirmed] Fennec Bugs
* Create Litmus testcases for bugs flagged as [https://bugzilla.mozilla.org/buglist.cgi?cmdtype=runnamed&namedcmd=in-litmus%3F%20Mobile in-litmus?]
* [https://wiki.mozilla.org/QA/TestDay/ TestDay] Promotions within Mobile Community Channels
* Write [http://wiki.mozilla.org/QA/Fennec/TechDocQueue technical documentation] to perform quality execution
 
=== Performance Benchmarking ===
* Scratch pad of [http://etherpad.mozilla.com:9000/3nS0MfR7yx notes]
* [[QA/Fennec/PerformanceBenchmark|Performance Testplan]]
 
=== Website Compatibility Testing ===
* Scratch pad of [http://etherpad.mozilla.com:9000/3nS0MfR7yx notes]
* [[QA/Fennec/WebsiteCompatibility|Web Compatibility Testplan]]
 
=== Sync Performance Instrumentation ===
Fennec sync continues to cause performance issues on first time syncing with very large amounts of data.  How you can help, is to provide sync logging through a custom extension, and submit your data to the Mozilla Services team.  Detailed instructions can be found [[QA/Fennec/Sync_Performance_Collection|here]].
<p>
More historical information on the project:
* Philikon's [https://wiki.mozilla.org/User:Philikon/SyncInstrumentation Sync Instrumentation Research]
* Blog on [http://philikon.wordpress.com/2011/03/09/javascript-perf-avoid-creating-objects-2/ profiling Javascript Objects]


== How You Can Contact Us ==
== How You Can Contact Us ==
If you're interested in helping out with anything below (or want to help with something else), please contact [http://quality.mozilla.org/user/21/contact jmaher]  or [http://quality.mozilla.org/user/1342/contact aakashd]. If you want to talk to the Mozilla Mobile Community, you can find us in our [http://widget.mibbit.com/?settings=1b10107157e79b08f2bf99a11f521973&server=irc.mozilla.org&channel=%23mobile IRC Channel].  
If you're interested in helping out with anything below (or want to help with something else), please contact us. We hang out on irc.mozilla.org in the channel #qa.
 
* Mailing Lists
* Mailing Lists
** mobile@mozilla.com
** [https://mail.mozilla.org/listinfo/android-mobile-nightly-testers Firefox for Android (Mobile Nightly Testers)]
* Newsgroups
* Newsgroups
** [http://groups.google.com/group/mozilla.dev.platforms.mobile/topics mozilla.dev.platforms.mobile]
** [http://groups.google.com/group/mozilla.dev.platforms.mobile/topics mozilla.dev.platforms.mobile]
** [http://groups.google.com/group/mozilla.dev.quality/topics mozilla.dev.quality]
** [http://groups.google.com/group/mozilla.dev.quality/topics mozilla.dev.quality]
* 3rd Party Testing (Waverley)
 
** [https://wiki.mozilla.org/QA/Fennec/Waverley Execution testplan] for Smoketests & BFTs
=== Download Firefox for Android ===
The easiest and most simplest way to help us by using Firefox for Android. One can download Firefox for Android on [https://play.google.com/store/apps/details?id=org.mozilla.firefox Google Play] and installing it on your Android device.
 
=Resources=
==== Reference Links ====
* This is a list of all the features being considered for development across products in the near future: https://wiki.mozilla.org/Features.
** To be specific this is the list of potential [https://wiki.mozilla.org/Features/Mobile mobile Firefox features] being considered.
* You can find a list of features to be shipped in the coming milestones: https://wiki.mozilla.org/Firefox/Flight_Tracking

Latest revision as of 19:25, 28 April 2016

Overview

The team focuses on testing current and upcoming releases of Firefox on Android. Some of our responsibilities include qualifying builds prior to a maintenance or milestone release. Some of the responsibilities include, but are not limited to:

  • Verifying bugs using a triage strategy
  • Running basic functional tests and smoke tests - mostly manual; but automated in the near future
  • Running our manual test suites on localized builds
  • Making sure web content is viewable, acceptable and ready for consumption in Firefox for Android

In addition to qualifying builds prior to a release, we perform ongoing tasks on the development branches to ensure no major problems get uplifted as we merge changes to the next branch. For example, on a daily basis we:

  • Confirm new unconfirmed bugs
  • Verify bugs on development branches like Aurora and Beta
  • Engage community and new volunteers in helping improve Firefox through test day events
  • Interact with developers to help them test the features they implement
  • Perform exploratory testing on new features while on the Aurora branch
  • Write test cases in Moztrap for new features
  • Monitor crash-stats to assess stability
  • File and track new crasher bugs as they appear in crash-stats
  • Manage web compatibility testing through community involvement or crowdsourcing companies that help in testing different Android device configurations and top-site testing

All of which help assess the quality of individual new features as well as the Firefox version as a whole.

Team Details

The team is lead by both Kevin Brosnan and Aaron Train who coordinate testing of their respective areas among several MoCo employees and community volunteers. Traditionally new features are tested by several MoCo QA employees. Anyone is welcome and encouraged to participate.

Team Members and Assignments

Name Contact Availability Project Assignments
Aaron Train atrain@mozilla.com MoCo Employee (full time) Lead - New features testing coordination.
Kevin Brosnan kbrosnan@mozilla.com MoCo Employee (full time) Lead - New features testing coordination.

QA Current Release Test Plans

Project Test Plan Description Scope Timeframe
Firefox for Android 31 QA activities revolving around Firefox Aurora in the time frame for Firefox 31 final release. Features in Firefox for Android 31 Upcoming Releases
Firefox for Android 32 QA activities revolving around Firefox Nightly in the time frame for Firefox 32 final release. Features in Firefox for Android 32 Upcoming Releases
Firefox for Android 33 QA activities revolving around Firefox Nightly in the time frame for Firefox 32 final release. Features in Firefox for Android 33 Upcoming Releases
Feature Ownership Testing, Planning, and Bug Triage of new and stable Firefox features used till Firefox for Android 30

Community Contribution

Anyone can participate. There are several ways in which YOU can participate:

  • File bugs
  • Triage bugs (confirm existing bugs and assign them to the right buckets)
  • Help test new features
  • Write test cases
  • Plan new features testing
  • Help others who want to get involved.

How You Can Contact Us

If you're interested in helping out with anything below (or want to help with something else), please contact us. We hang out on irc.mozilla.org in the channel #qa.

Download Firefox for Android

The easiest and most simplest way to help us by using Firefox for Android. One can download Firefox for Android on Google Play and installing it on your Android device.

Resources

Reference Links

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Fennec&oldid=1130225"