Labs/Test Pilot/New Tab Study

From MozillaWiki
Jump to navigation Jump to search

Introduction

Newtabstudy-thumbnail.png

This is a study on the effects of a new tab design on users' browsing behaviors. We will be doing a user study through Test Pilot. The study will first establish a baseline for user behavior, and then the next phase is A/B split testing.

In the A/B split testing phase, participants will be randomly divided into two groups with either (A) a blank new tab or (B) a new tab with most-visited sites (hereby known as "fancy tab"). The comparison between two groups of users will help us understand which type of new tab design can provide better browsing experience.

Goal

We want to know the benefits and drawbacks of a fancy tab page through full comparison between the blank tab page and the new one. The final results will be served as a reference for the work of UX design team on the new tab design.

Timeline

Tentative:

  • 05/31 - 06/03 Problem defined
  • 06/05 - 06/10 Confirm research questions and basic measures
  • 06/12 - 06/17 Implementation of both the study and the fancy tab interface
  • 06/19 - 06/24 More implementation refinement and testing
  • .......

Process

Research Questions

  1. Does the fancy tab slow users down (e.g., by distracting them), or speed them up (e.g., help them find the target site faster)? (time)
  2. Does the fancy tab discourage breadth in visited sites? (unique sites, frequency, distribution)
  3. How do users navigate to a website after they open a new tab? (location bar, search bar, top sites, bookmarks, history, etc.)
  4. Are there users who are more mouse-based and some who are more keyboard-based? How does a fancy tab affect them?

Survey Question

May launch a survey after the new tab study and ask questions explicitly.

Possible survey questions:

  • Do you mainly use the keyboard more or the mouse to navigate?
  • When you open a new tab, do you usually have a destination in mind?
  • How many websites do you mainly visit?
  • How familiar you do consider yourself with keyboard shortcuts?
  • How often do you open a new tab in order to visit a website?

Measures

All features and events that we would like to detect and record.

Current implementation without A/B options:

  • DB schema (
    • type_double timestamp,
    • type_double tab_id, // Same as the timestamp when the tab is created
    • type_string event,
    • type_string method, // Method that triggered the event
    • type_string url, // It should be hashed later
    • type_string clipboard, //will be removed later
    • type_int is_clipboard_url

)


  • EVENTS code:
    • start
    • navigation
    • leave
  • METHODS code:
    • unknown
    • -----------------------start
    • command_t
    • plus_btn
    • double_click
    • file_menu
    • ------------------------ navigation / cancelled_navigation
    • urlbar_enter
    • urlbar_go_btn
    • urlbar_drop_click
    • urlbar_drop_enter
    • urlbar_drop_button
    • search_enter
    • search_go_btn
    • search_drop_enter
    • search_drop_click
    • bookmark_bar
    • bookmark_menu
    • bookmark_top_sites
    • history_menu
    • ------------------------leave
    • leave
    • close

Implementation

Probably need to collaborate with UI design team, especially on the fancy tab interface design and implementation.

Results

Analysis

The screenshot of the collected data

With the user data collected by the study, we can measure:

  1. Domains
    1. measure entropy of the set of domains for each user --> distribution of the browsing entropies
    2. number of unique domains
    3. number of total browsing domains
    4. frequency of each domain
  2. Preferences
    1. frequency of each EVENTS.METHOD
    2. compare the keyboard-based methods and the mouse-based methods
  3. Timing
    1. time between the new tab open and the first page load
    2. time spent on each new tab before the user leaves or closes it
    3. time spent on each new page before the user leaves the tab or changes to another page
  4. Clipboard
    1. percentage of copying an url string before opening a new tab

Conclusion

Useful References

  1. HOW TO: Run Your First A/B Test in 4 Simple Steps
  2. Some tips to make a better A/B test
  3. Why A/B or Multivariate Testing Could Be The Easiest Way To Get More Out Of Your Website
  4. What Is Multivariate Testing, And Why It Is Superior To A/B Split Tests?
  5. Things you must know before you start A/B & Multivariate testing
  6. some downsides of A/B testing and multivariate testing
  7. A/B test or multivariate test?
  8. [1]
  9. http://etherpad.mozilla.com:9000/newtabABCDstudy
Retrieved from "https://wiki.mozilla.org/index.php?title=Labs/Test_Pilot/New_Tab_Study&oldid=326559"