Start-up Performance Improvements/TestPlan

From MozillaWiki
Jump to: navigation, search

Overview

Feature Release Target Dev Lead QA Lead Dev Status QA Status Health
Start-up Performance Improvements Firefox 15 Brian Bondy Jason Smith Landed In Progress OK

Feature Summary

This feature focuses on minimizing prefetch usage during firefox startup to improve startup performance of firefox. This is done through detecting that the Firefox PF read and write files on a Windows OS are not 0 KB. When they are detected to be 0 KB, we then write over those Firefox PF prefetch files to minimize the prefetch loading of those files on startup to a 0 KB PF file that is read only. This happens as a one-time operation on the first startup after 1 minute or during a firefox update from a firefox version that did not support this feature. Once the 0 KB PF file read only specification is set for the Firefox PF files, the 0 KB PF files should remain as 0 KB PF files that are read only. Additionally, the about:config parameter "app.update.service.prefetchCleared" is set to true, indicating that the prefetch process is finished.

Testing Scope

Major Features

  • Firefox start-up on Windows XP, Vista, and 7 32/64 bit

Software Qualities

  • Performance

Edge Cases

  • Fresh install of Firefox on a new computer (no prefetch files established)
  • Update from a firefox version without this feature to a firefox version with this feature
  • Firefox usage that affects the prefetch PF files outside of this feature (needs clarification)
  • Session restore with websites that typically result in re-load of fonts, plugins, etc
  • Testing start-up on under-performing harddrives
  • Startup performance while an anti-virus is scanning or a back-up utility is backing up files
  • Restarting use of firefox after windows sleep mode
  • Startup of firefox after starting up a windows machine
  • Crash firefox before write on Firefox PF files
  • Mozilla maintenance service being uninstalled while firefox is installed
  • State of pref vs. whether prefetch actually happened

Testing Strategy

Testing of this feature will be done both from a functional perspective to ensure that the implementation specified works as expected and from a metrics perspective to analyze how much better or worse start-up performance is as a result of this feature in various cases.

Sign Off Criteria

Aurora

  • Confirmed that the feature has landed (prove that the implementation has landed from bare bones testing)
  • No significant performance regressions in start-up performance time due to this feature
  • No severe regressions (e.g. deterministic crashes) due to this implementation

Beta

  • Functionality of this feature works as expected at the smoke and feature signoff level of test cases
  • No performance regressions in start-up performance time due to this feature
  • Data-driven proof that start-up performance improves on Windows in certain cases as a result of this feature (i.e. prove it's a value-added implementation)
  • No functional regressions in general implementation of desktop firefox due to this implementation

Release

  • Same as Beta

Test Case Management

Test cases for this feature shall be tracked in MozTrap. In MozTrap, they will be flagged as a Desktop Firefox 15 feature in the product version tag. Additionally, smoke tests will be flagged with the "smoke" tag. Feature signoff test cases will be flagged with the "feature signoff" tag. Each of these test cases are broke up into two categories: Functional and Performance. Functional covers test cases for verifying that the implementation matches the intended specification of the feature of what needs to be done. These test cases shall be flagged with the tag "functional." Performance covers test cases that run a particular test scenario, captures some start-up performance data, and measures it against a set of acceptance criteria. These test cases shall be flagged with the tag "performance." Example acceptance criteria could be seeing that the performance of some piece of data over multiple runs did not regress significantly and performance of metric X improved by 5% when running a certain test before and after this feature was implemented.

Ideas for test cases will be tracked here.

Hardware Requirements

  • Acer netbooks in the lab running Win 7 (under-performing)
  • Work machine & other machines in test lab containing windows OSes (typical scenarios)

Performance Metrics and KPIs

  • 50% improvement on the following metric:
    • SIMPLE_MEASURES_MAIN
  • Significant impact on the following metrics:
    • SIMPLE_MEASURES_CREATETOPLEVELWINDOW
    • SIMPLE_MEASURES_DELAYEDSTARTUPFINISHED
    • SIMPLE_MEASURES_DELAYEDSTARTUPSTARTED
    • SIMPLE_MEASURES_FIRSTLOADURI
    • SIMPLE_MEASURES_FIRSTPAINT
    • SIMPLE_MEASURES_SESSIONRESTORED
    • SIMPLE_MEASURES_SESSIONRESTOREINITIALIZED
    • SIMPLE_MEASURES_SESSIONRESTORERESTORING
    • SIMPLE_MEASURES_STARTUPCRASHDETECTIONBEGIN
    • SIMPLE_MEASURES_STARTUPCRASHDETECTIONEND
    • SIMPLE_MEASURES_STARTUPINTERRUPTED
    • SIMPLE_MEASURES_UPTIME
  • Histogram telemetry for SIMPLE_MEASURES becomes more positively skewed
  • Evolution telemetry for SIMPLE_MEASURES trends towards KPIs defined for each metric

Bugs

Open Bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Bugs Needing Verification

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Important References