This page describes a simple test to determine if the high bounce rate on the SUMO inproduct start page compared to the normal start page is because of the article links under the search box. The test is part of a larger project to optimize the SUMO start pages.

Overview

People visiting Firefox Support (SUMO) from Firefox itself (aka from "inproduct") are more likely to just leave the website without interacting with it, compared to people visiting the site from www.mozilla.com.

Another reason why we're seeing this difference could be because the two groups are actually ending up on different start pages. The inproduct start page lists simple tutorial and references, in an assumption that people visiting help from inproduct are more likely to just want to learn more about the browser, and not actually solve a problem.

This test aims to determine whether that assumption is true or not.

There are two main ways to reach the SUMO website:

Inproduct Start Page

First, there’s a support option within the Firefox browser itself (i.e., inproduct). Users clicking on "Help" from the Firefox menu are taken to the inproduct start page: http://support.mozilla.com/en-US/kb/Firefox+Help.

Below the search box, this start page lists references and tutorials that are originally coming from Firefox 2. This is based on the assumption that people using help from the product itself are more interested in learning about the program rather than solving a problem.

This start page has an average bounce rate of 85%. Among the people who stay on the page, 71% search.

Normal Start Page

The other way for users to enter SUMO is through the main navigation bar at www.mozilla.com. Users clicking on “Support” (depending on the localized version) go directly to the normal start page: http://support.mozilla.com/

Below the search box, this start page lists the most popular support articles, and a blurb for new users, listing articles useful for new users.

This start page has a bounce rate of 53%. Among the people who stay on the page, 50% search.

The A/B Test

Are users that are accessing inproduct help really more interested in reading references and tutorials about Firefox, or are they also trying to solve a support problem?

That is the basic question this test tries to answer. Since a larger percentage of the visitors interact with the site on the normal start page (which features lists of the most popular support articles), this test will expose a group of people visiting SUMO from inproduct to a special start page with the article lists copied from the normal start page.

• A copy of the inproduct start page will created.
  • URL: http://support.mozilla.com/en-US/kb/Firefox+Help+1
• A second copy of the inproduct start page will be created (here called the hybrid start page), that looks like the inproduct start page, but with the article lists below the search box copied from the normal start page.
  • URL: http://support.mozilla.com/en-US/kb/Firefox+Help+2
• We will limit the test group to the en-US locale.
• We would then split the visitors of the inproduct start page between the inproduct start page and the hybrid start page.

Measuring Results

The key metrics to compare in this test are:

• Bounce rate -- This is the main metric we want to compare. If the hybrid start page has a lower bounce rate, that should mean that the normal start page is more in line with what visitors of SUMO expect to see.
• Search ratio -- If the number of people searching (rather than interacting/clicking on links) drops on the hybrid start page, that should mean that the provided article links are more useful. Of course, this is not true if the bounce rate above increases at the same time.