MozillaQualityAssurance:Environments

This page explores some of the topics around creating Firefox test environments that mimic "real world" and "typical" end user configurations.

Overall Goal

Configure the Mozilla testing systems as close as possible to a "typical" Firefox user in order to discover problems early and provide better configuration coverage.

Every user does not install Firefox into a fresh copy of Windows XP SP2 and people do not always upgrade to the latest and great MSFT patches. Their profiles may have been used for years and they have their web cache full of data or run a bunch of extensions. The goal of this exercise is to come up with a reasonable set of configs that best represent some typical configurations of our users.

Our current testing methodology uses a (random? chaotic?) combination of configuration environments that are not documented, controlled or reproducible.

Requirements

• Scope out several "typical end user configurations" for Firefox.
• Offer an easy way to do testing and throw away the environment
• Keep the base environments up to date
• Easily "apply a specific Firefox config" against the base environment
• Some of the parameters to consider include:
  • Operating System (clean install vs. existing installation), version number, patches, etc... - Windows XP, Vista, Mac, Linux (which one?)
  • OS Languages (e.g. are they running US Windows XP or Japanese XP)
  • Firefox (clean new install), no profile versus Existing Profile
  • Profile character:
    • Clean web cache vs. 125 MB worth of stuff
    • 2 vs. 15 addons (which ones to select?)
    • Selection of saved passwords?
    • Other profile aspects?
  • One versus multiple OS users? (not sure)
  • One versus multiple Firefox profiles? (not sure)

Recommendation

• Develop a "minimum set" (3-4 configs) and an "expanded set" (up to 10?)
• Use our Vmware server and "snapshot capabilities" to define the following sets of VM's
• Typical Profiles
  • No Profile (ie, new user to Firefox)
  • Profile A: 120 MB worth of cached web pages, 3 addons [typical]
  • Profile B: 200 MB worth of cached web pages, non-default home page, 10 addons [stress test]
• Typical OS Setups
  • OS Setup 1: Windows XP SP2 + all high priority WindowsUpdate patches
  • OS Setup 2: Windows XP SP2 + no additional patches
  • OS Setup 3: Mac OS X 10.4.9 + all available SoftwareUpdate patches
  • Minimum Set:
    • Config 1: OS Setup 1 + no Firefox Profile
    • Config 2: OS Setup 1 + Firefox 2.0.0.x + Profile A
    • Config 3: OS Setup 1 + Firefox 2.0.0.x + Profile B
    • Config 4: OS Setup 3 + Firefox 2.0.0.x + Profile A
  • Expanded Set:
    • Config 5: OS Setup 3 + Firefox 2.0.0.x + Profile B
    • Config 6: OS Setup 3 + no Firefox Profile
    • Config 7: OS Setup 2 + no Firefox Profile

• Create VM instances for OS Setup 1 sn
• Use the VM instances in "read only mode" to test
• VM Instance maintenance: Periodically, update the base config with the latest Microsoft Windows patches

• Our Vmware Server allows us to store multiple copies of vm instances. So, perhaps we create several base instance (e.g. Windows XPSP2, Windows XP SP2+all current patches) and then script the creation of getting these base instances to the configs that we want (e.g. loading them with profile data, etc...).
• For Mac based testing of Window Firefox, you can do the same thing but using Parallels. Create a clean Windows XPSP2 instance, save the config and copy it to a file server location. Using a copy of that instance, create Windows XPSP2+patches, save that instance. When testing Firefox, use a copy from the fileserver and then throw away the instance.
• Keep the base instances updated with the latest patches from Microsoft periodically.
• For Mac testing, not certain how we do that.

Notes from Profile and Configuration Meeting Held on 6-13-07

Typical OS Setup:

Tier 1:

Windows XP with latest updates Windows Vista with latest updates Mac OSX with latest updates Linux - which distro? Needs more investigation

• First state implementation will focus on Tier 1 only

Tier 2:

XP SP2 - No patches

	Profile A	Profile B	Profile C	Profile D
Cache	120	200
History
Bookmarks
Downloads
Form Data
Passwords
Addons
Tabs
Non Default Homepage
Plugins (Java, Flash, etc)
Search Plugins
Installation History
Multi Users
Preferences

1Cache 120 MB 200 MB 1History 1Bookmarks 1Downloads 1Form Data 1Passwords 1Extensions and Themes 3 50 2Tabs 2Non Default Homepage 1Plugins (Java, Flash) 3Search Plugins 2Installation History - Clean Install/Paveover Install/Upgrade 2Multi-Users User 2 users Admin/Non Admin 2+Users 3Preferences

Profiles

• A No Profile
• B Typical Profile
• C Large Profile
• D Stress Profile

Configurations

• 1 OS1 - FF 2.0.0.x - Profile A (Smoke)
• 2. OS1 - FF 2.0.0.x - Profile B (BFT)
• 3. OS1 - FF 2.0.0.x - Profile C (Smoke)
• 4. OS2 - FF 2.0.0.x - Profile B (Smoke)
• 5. OS3 - FF 2.0.0.x - Profile B
• 6. OS4 - FF 2.0.0.x - Profile B

Other Issues

Locales: Admin

Priorities: Magnitude of problem (data loss) Common Compatibility (Problems with Java)

Next Steps: