Overview

The load testing of the weave services, will rely on synthetic load generation against the server with concurrent live testers performing user sync operations and measuring the sync response times and ensuring basic weave operations are performing well. The live testers will be deployed over a number of geographical locations to measure the actual response times from a number global locations.

The test for the live testers to perform will be conducted in 2 phases. The 1st phase is to get systems initialized, create user synchronized data like history and bookmarks and to establish baseline performance sync measurements under light load server conditions. The second phase will be a rerun of the performance test under saturated high load levels. Performance measurements under noload and loaded conditions will compared and the difference will be checked against expected deltas.

Perquisites

• 10 testers located evenly in serveral geographic locations (north/south america, europe, africa, asia, australia)
• (ideally) Tester should have access to 2 different firefox supported platforms (linux, win7, mac OSX 10.6). Either physical platforms (preferred) or VMs.

Live User Initialization

• Note for testing Monday, May 24
  • There is a new client build (1.3 build 7) available
    • you can update your existing version Tools->Add-ons->Find Updates
      • Follow instructions to update the Firefox Sync(Weave addon)
    • The new build can also be installed in a fresh browser instance noted below at this location
      • If creating a user login, we still need to use the custom server option using the following url: phx-auth.services.mozilla.com
  • Desired testing
    • Exploratory testing around sync operations using existing login
      • sync local, sync remote, sync merge
      • ensure that all edits updates to one instance are properly updated with another sync instance
      • sync data to test and ensure proper updates on all instances
        • history
        • tabs
        • passwords
        • preferences
    • Exploratory testing around sync operations with new account login
      • Same testing as above, but starting with the creation of a new user login on the server.
      • Tim Evans will coordinating the testers who will use a new account and those who should use their existing account.

1. Install version 3.5.10 Firefox browser on one system and 3.6.4 version of the Firefox browser on the other. The download locations are ftp sites where builds for each supported platform and locale reside. Choose the appropriate build for your platform and locale. For example: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1/win32/en-US/ is the location of the win32 3.5.10 build for English US.
   1. Firefox 3.5.10 download is located http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.5.10-candidates/build1
   2. Firefox 3.6.4 download is located http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.4-candidates/build3
2. startup the browser with a clean profile on each system
   1. If you are a firefox user, it will be important not to use your personal data for testing.
   2. Instructions for starting the browser with a clean profile is located here
3. Install the latest weave sync addon in each of the browsers
   1. Install the latest weave sync client from this location
   2. Register with the weave services
   3. Instructions for installing and registering with the weave service is located here
4. On one of the Firefox browsers start creating sync data
   1. Browse to many sites
   2. Add lots of tabs
   3. Bookmark a number of urls
      1. add tags
      2. add toolbar bookmarks
   4. add login/passwords to pass protected sites (encrypted)
5. On the other system
   1. Login into weave service a sync down the server data captured from session above.
      1. Ensure synchronized data is duplicated on 2nd system.

Tests to Perform

Phase 1

As mentioned above, the load test will be done in 2 phases. The first phase is to install firefox and create the user data followed by [1] Functionality Smoke Tests and then [2] Sync Performance Tests. After the first phase, user should save their user profile data.

Phase 2

The second phase will be conducted several days later when the weave systems is at the full load capacity. In the second phase the tests will be conducted in reverse order [2] Sync Performance Tests and the [1] Functionality Smoke Tests. Phase 2 should begin with the reload of the saved profile data.

[1] Functionality Smoke Tests

• With both systems weave data synchronized, ensure user data update operations are reflected in the secondary browser instance after a weave sync:
  • data deletion (tabs, bookmarks,history,passwords)
  • data addition (tabs,bookmarks, history,passwords)
  • data change (tabs,bookmarks,history,passwords)
  • data merge (bookmark tags)

[2] Sync Performance Tests

• weave delete local and download weave server data synchronization performance test
  1. clear activity log
  2. for 5 times do
     1. Tools->Weave->Preference->Manage Account->Reset Sync->Replace all data on this computer with your Weave data
  3. save activity log as <user-name>_<phase>_sync-replace-local.log
• weave delete weave server data and upload local data synchronization performance test
  1. clear activity log
  2. for 5 times do
     1. Tools->Weave->Preference->Manage Account->Reset Sync->Replace all other devices with your local data
  3. save activity log as <user-name>_<phase>_sync-replace-server.log

Results Reporting

• The main reporting of user activity will be the Activity logs
• report any perceived errors (data loss, data corruption, http errors etc).

References

• Overall weave client testplan: https://wiki.mozilla.org/QA/Weave_Test_Plan/1.3
• Build location: https://services.mozilla.com/sync/install.php
• Litmus Smoketests: https://litmus.mozilla.org/run_tests.cgi?test_run_id=83

• encourage people to search this list before filing a bug.
  • Known Weave bugs (minus dupes): http://bit.ly/btftoZ
  • Known Weave FIXED bugs: http://bit.ly/bxx7CT
  • Known Weave 1.3 final blockers: http://bit.ly/avVwal

• knowledge base articles
  • http://support.mozilla.com/en-US/kb/Profiles
  • http://support.mozilla.com/en-US/kb/Managing+profiles