QA/Fennec: Difference between revisions

Revision as of 20:34, 8 August 2011

Home of those who want to Propel Fennec Quality!

Milestones

Current Releases

8.0: Testplan
7.0: Testplan
6.0: Testplan

Past Releases

5.0: Testplan
2.0: Testplan
1.1: Testplan
1.0: Testplan

Planning

2011
~~2010~~

What We Do

Testing Strategy

Manual

Daily Smoketests on the latest [nightly.mozilla.org/mobile.html Nightly build] on the current development branch and trunk
- Verificiations of overnight resolved bugs on the current development branch
- Run BFT(s)/Focused testing around areas that area affected by resolved bugs
Weekly triage of new and unconfirmed bugs
Bi-Monthly Testdays

Automation:

* Unittests on Fennec
  1. Mochitests on Fennec
  2. Reftests on Fennec
* For more information, take a look at the TestDev Wiki for Mobile

Unique Areas of Testing:

* Test Guide on Daily Browsing
* Website Compatibility
* Device Integration Checklist
* OS Integration Checklist
* Updates Testing

Release Testing

Mobile Release Checklist
Template for results tracking page

Release	Smoketest	BFT	Blocker Verifications	L10n	Website	Updates
Early-Stage Betas	YES	NO	YES	1 RTL, 1 non-Latin, 1 Multi-L10n Switch	NO	YES
Late-Stage Betas (3+)	YES	YES	YES	1 RTL, 1 non-Latin, Full Multi-L10n Switch	NO	YES
Major Releases	YES	YES	YES	2 RTL, 2 non-Latin, Full Multi-L10n Switch	YES	YES

Fennec Crash Reporting

What We Use

Tools

Litmus

Fennec Test Runs

Device Anywhere

DeviceAnywhere

Performance Graphs

Performance Graphs

Resources

Device Anywhere Tutorial

Device Anywhere Tutorial

Mobile (HG) Repository

mozilla-central

Mobile Source Code

Fennec MXR Source Code Web-Based Viewer

Builds

Where Do We Need Help

Test Tool Development

Write automated testcases
Port Automation Frameworks to WinCE
Find, Fix, Verify UnitTest Failures

Execution

Triage Unconfirmed Fennec Bugs
Create Litmus testcases for bugs flagged as in-litmus?
- Fennec Component Ownership in Litmus
TestDay Promotions within Mobile Community Channels
Write technical documentation to perform quality execution

Performance Benchmarking

Scratch pad of notes
Performance Testplan

Website Compatibility Testing

Scratch pad of notes
Web Compatibility Testplan

Sync Performance Instrumentation

Fennec sync continues to cause performance issues on first time syncing with very large amounts of data. How you can help, is to provide sync logging through a custom extension, and submit your data to the Mozilla Services team. Detailed instructions can be found here.

More historical information on the project:

Philikon's Sync Instrumentation Research
Blog on profiling Javascript Objects

How You Can Contact Us

If you're interested in helping out with anything below (or want to help with something else), please contact jmaher or aakashd. If you want to talk to the Mozilla Mobile Community, you can find us in our IRC Channel.

Mailing Lists
- mobile@mozilla.com
Newsgroups
- mozilla.dev.platforms.mobile
- mozilla.dev.quality
3rd Party Testing (Waverley)
- Execution testplan for Smoketests & BFTs

@@ Line 24: / Line 24: @@
-=== Testing Strategy ===
+=== Testing Strategy  ===
+; Manual
+* [[QA/Fennec/Waverly|Daily Smoketests]] on the latest [nightly.mozilla.org/mobile.html Nightly build] on the current development branch and trunk
+** Verificiations of overnight resolved bugs on the current development branch
+** Run BFT(s)/Focused testing around areas that area affected by resolved bugs
+* Weekly triage of [http://bit.ly/qbtF9O new and unconfirmed bugs]
+* [http://quality.mozilla.org/ Bi-Monthly Testdays]
-Manual:
+Automation:
- * [[QA/Fennec/Waverly|Daily Smoketests]] on the latest nightly build on the current development branch and trunk
-. Verifications of overnight resolved bugs on the current development branch
-. Run BFT(s)/Focused testing around areas that are affected by resolved bugs
- * Weekly triage of feedback received via the Fennec Reporter. URLs to the current lists can be found here: [http://reporter.mozilla.org/app/query/?report_description=&report_useragent=&report_gecko=&report_language=&report_platform=&report_oscpu=&report_product=Fennec%2F1.1a2pre&report_file_date_start=&report_file_date_end=YYYY-MM-DD&show=25&host_hostname=&report_problem_type=-1&report_behind_login=-1&&submit_query=Search 1.1a2pre] and [http://reporter.mozilla.org/app/query/?report_description=&report_useragent=&report_gecko=&report_language=&report_platform=&report_oscpu=&report_product=Fennec%2F1.0.0&report_file_date_start=&report_file_date_end=YYYY-MM-DD&show=25&host_hostname=&report_problem_type=-1&report_behind_login=-1&submit_query=Search 1.0.0]
- * Weekly triage of filed bugs and those that are nominated and in a New state
- * Before string freeze, update [https://litmus.mozilla.org/run_tests.cgi?test_run_id=48 the String Guide] in the latest Localizer Test Run with new features added for that release
- * Bi-Quarterly Testdays
-Automation:
   * [https://wiki.mozilla.org/Mobile/Fennec_Automation Unittests on Fennec]
 . [https://wiki.mozilla.org/Mobile/Fennec_Mochitest Mochitests on Fennec]
 . [https://wiki.mozilla.org/Mobile/Fennec_Reftests Reftests on Fennec]
   * For more information, take a look at the [https://wiki.mozilla.org/Mobile/Fennec_TestDev TestDev Wiki for Mobile]
 Unique Areas of Testing:
   * [http://wiki.mozilla.org/QA/Fennec/Daily_Browsing Test Guide on Daily Browsing]
   * [http://wiki.mozilla.org/QA/Fennec/Webpages Website Compatibility]