QA/Automation/Projects/Endurance Tests: Difference between revisions

From MozillaWiki
< QA‎ | Automation‎ | Projects
Jump to navigation Jump to search
 
(28 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= Overview  =
__NOTOC__


{| class="fullwidth-table"
==Overview==
|-
We're exploring different ways and or abilities to utilize [https://developer.mozilla.org/en/Mozmill Mozmill] and its testing capabilities to determine if Firefox can sustain the continuous expected load seen through simulated 'real' web usage. These tests will ultimately create a simple real world model that employs a high but normal load pattern of actions for the browser in order to determine any potential:
| style="width:30%" |  
 
| style="width:70%" |  
* Browser performance degradation
|- valign="top"
* Crashes
| '''Lead:'''  
 
| [mailto:atrain@mozilla.com Aaron Train]
Initially, this project will focus on setting up an environment employing a new test harness that will loop through simple test snippets to expose the above areas of concern within Firefox.
|- valign="top"
 
| '''Co-workers:'''  
{| style="width: 90%; margin: 0 0 1em 1em" |
| [mailto:dhunt@mozilla.com Dave Hunt]
| style="width: 20%" |
|- valign="top"
| style="width: 80%" |
| '''Dates:'''  
|- valign="top"
| Kick-off: December 2010
| '''Name:'''
Phase I: December 2010 (goal)
| Endurance Tests
|- valign="top"
|- valign="top"
| '''Status:'''  
| '''Leads:'''
| In process of requirements gathering
| [mailto:dhunt@mozilla.com Dave Hunt], [mailto:ahughes@mozilla.com Anthony Hughes]
|- valign="top"
|- valign="top"
| '''Discussion:'''  
| '''Contributors:'''
| [http://etherpad.mozilla.com:9000/KabTTn3qnA Etherpad Document]
| [mailto:atrain@mozilla.com Aaron Train], [mailto:hskupin@mozilla.com Henrik Skupin], [mailto:gmealer@mozilla.com Geo Mealer]
|- valign="top"
|- valign="top"
| '''Repository Location:'''
| '''Tracking Bug(s):'''
| TBD
| {{bug|629065}}
|- valign="top"
|- valign="top"
| '''Tracking Bug(s):'''
| '''Discussions:'''
| TBD
| [http://etherpad.mozilla.com:9000/KabTTn3qnA General], [http://etherpad.mozilla.com:9000/mLWlTg7DwK Test Snippets]
|- valign="top"
| '''Documentation:'''
|
|}
|}


= Summary =
== Goals for Q3/11 ==


We're exploring different ways and or abilities to utilize [https://developer.mozilla.org/en/Mozmill Mozmill] and its testing capabilities to determine if Firefox can sustain the continuous expected load seen through simulated 'real' web usage. These tests would create a simple real world model that employs a high but normal load pattern of actions for the browser in order to  determine any potential:
{| class="fullwidth-table" |
| style="background:#EFEFEF; width: 15%" | Status
| style="background:#EFEFEF; width: 25%" | Task
| style="background:#EFEFEF; width: 60%" | Description
|- valign="top"
| {{done|}}
| Improve coverage
| Improve coverage of Endurance tests for real world use-case scenarios and add additional metrics to support the memshrink project for Firefox
|}


* Browser performance degradation
= Meetings =
* Crashes
{{:QA/Mozmill Test Automation/Endurance Tests/Meetings}}


= Definitions =
= Definitions =
Line 45: Line 50:
= Roadmap =
= Roadmap =


== Phase I - Planned Q4 2010 ==
== Phase I - Completed ==


# Command line wrapper for Mozmill that:
# Command line wrapper for Mozmill that:
#* Takes a JSON parameter with:
#* Takes the following parameters:
#** Number of times to loop
#** Number of times to loop
#** Maximum duration of test
#** Delay to wait before each iteration
#* Runs a single 'snippet' according to the above parameters
#* Runs a single 'snippet' according to the above parameters
#* Outputs memory usage to console
#* Outputs memory usage to the report
# A 'snippet' that uses the new command line wrapper and outputs results that would potentially expose degradation of performance
# A 'snippet' that uses the new command line wrapper and outputs results that would potentially expose degradation of performance
# New reports for endurance tests in Mozmill dashboard
== Phase II ==
# Determine achievable additional performance metrics
# Determine achievable client hardware details
# Restart between each test by default (override from command line)
# All tests must use local content
# Improve performance of charts with considerable data
== Phase III ==


== Phase ? ==
== Phase ? ==
 
* Create a view for results listed by memory consumption
* Gracefully handle the AUT crashing, and ensure all reports are intact
* Gracefully handle the AUT crashing, and ensure all reports are intact
* Output results to a location on the file system
* Output results to a location on the file system
Line 68: Line 85:
* Determining saturation points (baselines) in the AUT and monitor for major discrepancies
* Determining saturation points (baselines) in the AUT and monitor for major discrepancies


== Outstanding Items ==
==Resources==
 
* [[QA/Mozmill_Test_Automation/Endurance_Tests/Documentation|Documentation]]
# Endurance Automation Wrapper - This is the harness that will launch the AUT, run the test snippet until either the loop count is reached or the maximum time has elapsed, and report results.
* [http://blargon7.com/tag/endurance-tests/ Blog Posts]
# Initial Test Snippet - The intention is to write a simple test snippet that opens and closes the add-ons manager.
 
= Additional Requirements =
 
* Use debug nightly builds of Firefox to be able to get a full stack of information

Latest revision as of 10:40, 13 January 2014


Overview

We're exploring different ways and or abilities to utilize Mozmill and its testing capabilities to determine if Firefox can sustain the continuous expected load seen through simulated 'real' web usage. These tests will ultimately create a simple real world model that employs a high but normal load pattern of actions for the browser in order to determine any potential:

  • Browser performance degradation
  • Crashes

Initially, this project will focus on setting up an environment employing a new test harness that will loop through simple test snippets to expose the above areas of concern within Firefox.

Name: Endurance Tests
Leads: Dave Hunt, Anthony Hughes
Contributors: Aaron Train, Henrik Skupin, Geo Mealer
Tracking Bug(s): bug 629065
Discussions: General, Test Snippets

Goals for Q3/11

Status Task Description
[DONE] Improve coverage Improve coverage of Endurance tests for real world use-case scenarios and add additional metrics to support the memshrink project for Firefox

Meetings

From Q4/2011 the endurance tests project will report updates and discuss topics in the QA Automation Services regular meetings.

Meeting Notes

Definitions

Test Snippet
A test snippet will be one or more actions that can be performed within the AUT written in JavaScript according to the Mozmill specifications.

Roadmap

Phase I - Completed

  1. Command line wrapper for Mozmill that:
    • Takes the following parameters:
      • Number of times to loop
      • Delay to wait before each iteration
    • Runs a single 'snippet' according to the above parameters
    • Outputs memory usage to the report
  2. A 'snippet' that uses the new command line wrapper and outputs results that would potentially expose degradation of performance
  3. New reports for endurance tests in Mozmill dashboard

Phase II

  1. Determine achievable additional performance metrics
  2. Determine achievable client hardware details
  3. Restart between each test by default (override from command line)
  4. All tests must use local content
  5. Improve performance of charts with considerable data

Phase III

Phase ?

  • Create a view for results listed by memory consumption
  • Gracefully handle the AUT crashing, and ensure all reports are intact
  • Output results to a location on the file system
  • Read parameters in from disk
  • Determine areas of perceived risk within the AUT
  • Write more test snippets based on above
  • Gather metrics on more system resources
  • Present results in an easy to digest way
  • Automate the running of endurance tests
  • Abort tests that exceed predetermined resource thresholds
  • Determining saturation points (baselines) in the AUT and monitor for major discrepancies

Resources

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Automation/Projects/Endurance_Tests&oldid=887721"