QA/Automation/Projects/Endurance Tests: Difference between revisions
< QA | Automation | Projects
Jump to navigation
Jump to search
No edit summary |
|||
| Line 1: | Line 1: | ||
= Overview = | = Overview = | ||
{| class="fullwidth-table" | {|class="fullwidth-table" | ||
|- | |- | ||
| style="width:30%" | | | style="width:30%" valign="top" | '''Lead:''' | ||
| style="width:70%" | | style="width:70%" valign="top" | [mailto:dhunt@mozilla.com Dave Hunt] | ||
|- | |||
| [mailto:dhunt@mozilla.com Dave Hunt] | |||
|- | |||
| '''Co-workers:''' | | '''Co-workers:''' | ||
| [mailto:atrain@mozilla.com Aaron Train] | | [mailto:atrain@mozilla.com Aaron Train]<br />[mailto:hskupin@mozilla.com Henrik Skupin]<br />[mailto:gmealer@mozilla.com Geo Mealer] | ||
|- valign="top" | |- | ||
| '''Dates:''' | | valign="top" | '''Dates:''' | ||
| Kick-off: December 2010 | | Kick-off: December 2010<br />Phase I: Q1/2011 (goal) | ||
Phase I: | |- | ||
| valign="top" | '''Status:''' | |||
| | | Requirements gathering / planning | ||
| '''Status:''' | |- | ||
| | | valign="top" | '''Discussion:''' | ||
|- valign="top" | |||
| '''Discussion:''' | |||
| [http://etherpad.mozilla.com:9000/KabTTn3qnA Etherpad Document] | | [http://etherpad.mozilla.com:9000/KabTTn3qnA Etherpad Document] | ||
|- valign="top" | |- | ||
| '''Repository Location:''' | | valign="top" | '''Repository Location:''' | ||
| TBD | |||
|- | |||
| valign="top" | '''Tracking Bug(s):''' | |||
| TBD | | TBD | ||
|- valign="top" | |- | ||
| ''' | | valign="top" | '''Documentation:''' | ||
| TBD | | TBD | ||
|} | |} | ||
| Line 58: | Line 53: | ||
#** Maximum duration of test | #** Maximum duration of test | ||
#* Runs a single 'snippet' according to the above parameters | #* Runs a single 'snippet' according to the above parameters | ||
#* Outputs memory usage to | #* Outputs memory usage to file | ||
# A 'snippet' that uses the new command line wrapper and outputs results that would potentially expose degradation of performance | # A 'snippet' that uses the new command line wrapper and outputs results that would potentially expose degradation of performance | ||
Revision as of 17:59, 12 January 2011
Overview
| Lead: | Dave Hunt |
| Co-workers: | Aaron Train Henrik Skupin Geo Mealer |
| Dates: | Kick-off: December 2010 Phase I: Q1/2011 (goal) |
| Status: | Requirements gathering / planning |
| Discussion: | Etherpad Document |
| Repository Location: | TBD |
| Tracking Bug(s): | TBD |
| Documentation: | TBD |
Meetings
Summary
We're exploring different ways and or abilities to utilize Mozmill and its testing capabilities to determine if Firefox can sustain the continuous expected load seen through simulated 'real' web usage. These tests will ultimately create a simple real world model that employs a high but normal load pattern of actions for the browser in order to determine any potential:
- Browser performance degradation
- Crashes
Initially, this project will focus on setting up an environment employing a new test harness that will loop through simple test snippets to expose the above areas of concern within Firefox.
Definitions
- Test Snippet
- A test snippet will be one or more actions that can be performed within the AUT written in JavaScript according to the Mozmill specifications.
Roadmap
Phase I - Planned Q1 2011
- Command line wrapper for Mozmill that:
- Takes a JSON parameter with:
- Number of times to loop
- Maximum duration of test
- Runs a single 'snippet' according to the above parameters
- Outputs memory usage to file
- Takes a JSON parameter with:
- A 'snippet' that uses the new command line wrapper and outputs results that would potentially expose degradation of performance
Phase ?
- Gracefully handle the AUT crashing, and ensure all reports are intact
- Output results to a location on the file system
- Read parameters in from disk
- Determine areas of perceived risk within the AUT
- Write more test snippets based on above
- Gather metrics on more system resources
- Present results in an easy to digest way
- Automate the running of endurance tests
- Abort tests that exceed predetermined resource thresholds
- Determining saturation points (baselines) in the AUT and monitor for major discrepancies
Outstanding Items
- Endurance Automation Wrapper - This is the harness that will launch the AUT, run the test snippet until either the loop count is reached or the maximum time has elapsed, and report results.
- Initial Test Snippet - The intention is to write a simple test snippet that opens and closes the add-ons manager.
Additional Requirements
- Use debug nightly builds of Firefox to be able to get a full stack of information