QA/Activity Stream

From MozillaWiki
< QA
Jump to: navigation, search

Revision History

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
01/25/2016 1.0 Paul Oiegas Created first draft
02/16/2016 1.1 Paul Oiegas & Vlad Bacia Added general details form received documentation
02/17/2016 1.2 Paul Oiegas & Vlad Bacia Added the remaining details from received documentation
06/07/2016 1.3 Paul Oiegas & Vlad Bacia Updated all sections that were affected by the latest changes, till the launch of Activity Stream in Test Pilot.



Activity Stream is a broad effort to help users recover things they’ve found before, move easily between Firefox on different devices, and discover new things on the web. Trough this, we hope to increased engagement and users retention.

Test Pilot

We intend to use Test Pilot as the launch vehicle for the Activity Stream MVP. It will allow us to get into market quickly, solicit feedback from real users, and develop the product in a data-driven way. Test Pilot will also help us acquire new users for our test and eventually allow us to smoothly migrate those users to this feature-set in Firefox release.

Milestone 1

Enhanced Data

A key emphasis of Activity Stream is having richer data. This allows us to use more recognizable imagery and longer term allows us to use various page engagement signals to develop a deep understanding of what’s interesting to a user. We should begin by collecting the following data:

  • Favicons (improve retention)
  • Large favicons
  • Full text of the content (Readability)
  • Lead image URL from the full text
  • Fixed word summary (first couple of sentences)
Addon key features
  • Timeline

The timeline will be a new, core UI surface in Firefox. It is the eventual replacement for the current Library UI. It is a reverse chronological list of the most interesting pages you’ve visited and activities you’ve done in Firefox. This is a great way to interact with the content and tasks a user has interacted with over a short time horizon (48 hours). This also provides a single destination to find anything that you’ve done in the browser. Whether that be something as prosaic as visiting a site or something more nuanced as sharing a specific page with a specific user.

  • New Tab

New Tab is an existing UI surface, but it is currently underutilized. We will keep the existing search access point. We will optimize the current Top Site’s tiles in order to allow for other content on new tab. We will show 6 smaller, fixed top sites with logos of the sites, when available. We will introduce two new sections: Recent Activity and Spotlight. Recent Activity will be a summary view of your timeline and provide the most recent actions you’ve taken (Bookmarks for now) and other interesting history. Spotlight will be a space for us to feature “nearly now” content along with recommended content in the future.

Milestone 2


The Awesomebar is a critical part of the long term success of activity stream. Eventually we should replace frecency with a more nuanced understanding of interestingness to the user. Until then we should make minimal changes to the Awesomebar as the current frecency algorithm is very successful for many users. We should focus on improving recall rather than changing any core algorithm functions. This means better favicons and potentially other images as well.

Saving and Sharing

Our user research shows that people repeatedly share and return to content in the browser. We should embrace this by creating a flexible save and share toolbar menu. It should allow the user to share and save to the most common services such as Facebook or Pocket.

We can quickly provide this functionality on desktop by leveraging the existing “shareplane” and offer the most popular services as default options in our new saving and sharing menu. We should also include the existing features built into the browser such as “Email link...” and “Save to Pocket”. There should eventually be a “more” option at the end of this menu that connects the user with the built-in sharing options provided by their OS.

Android and iOS provide advanced sharing options at the system level and Firefox for Android and iOS both do a great job leveraging these systems. For the MVP we can rely on these existing integrations. In the future we should consider if we want to have Firefox specific share options on mobile.

Share and Save actions are high signal operations. The user is clearly indicating that something is important. We should flag shared URLs and add an activity entry to the timeline including as much detail as possible e.g. “Emailed this page to Bryan Bell”.


This wiki details the testing that will be performed for Activity Stream project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:

  • What will be tested
  • How testing will be performed


Technical Program Manager contact: Jenn Chaulk

Engineering contact: Tim Spurway

Production contact: Maria Popova

User Experience contact: Bryan Bell

Rares Bologa - PM for QA team

Paul Oiegas - leading QA efforts

Vlad Bacia - QA

Cosmin Muntean - QA

Ciprian Muresan - QA

Emil Pasca - QA

Feature email address: Activity Stream

Testing summary

Scope of Testing

In Scope

Activity stream is an add-on that will comprise saved bookmarks, saved pockets and more in one place and it will display all the user activity organized in an reverse chronological timeline. This will allow the user to limit the displayed pages or domains from his activity. Also the add-on will provide more searching options based on favicons, images and keywords from the read article (first sentences from it).

The scope of our testing is the Activity Stream-desktop add-on and its functionality. The testing effort for Activity Stream will be invested on the following areas:

  • integration: verify the add-on integration with the current browser functionalities and UI;
  • functionality: basic and advanced functionality to be verified according to the existing requirements;
  • usability: the Activity Stream timeline and New Tab content must be easy to use and straightforward;
  • testing how Bookmarks and History are imported in the add-on functionalities (from profile or Firefox account);

Out of Scope

- The mobile implementation and testing.

Requirements for testing


Testing will be performed on following OSes:

  • Windows XP (x32)
  • Windows Vista (x32 & x64)
  • Windows 7 (x32 & x64)
  • Windows 8.1 (x32 & x64)
  • Windows 10 (x32 & x64)
  • Ubuntu 14.04 (x32 & x64)
  • Ubuntu 15.04 (x32 & x64)
  • Fedora 23 (x64)
  • Mac OS X 10.9
  • Mac OS X 10.10
  • Mac OS X 10.11

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.

Ref Function Test Objective Evaluation Criteria Test Type Owners
1 Install Activity stream add-on Verify that the add-on is correctly installed Add-on UI changes are present in the browser Manual Eng Team
2 Timeline functionality and content
Removed for now
Verify all Timeline tab options and related pages content 1. Open the activity stream tab
2. Check "All" page content
3. Check "Bookmarks" page content
4. Check "Downloads" page content
5. Check "Tabs" page content
6. Check Search (Filter) functionality and other Timeline associated features
Manual Eng Team
3 New Tab functionality and content Verify that Activity Stream content is displayed in a new tab Activity stream user content is displayed in each new tab opened Manual Eng Team
4 Bookmarks & history Verify that profile / account bookmarks and history are correctly imported in the add-on All the previous stored data is displayed on the add-on pages Manual Eng Team
5 Share functionality
Removed for now
Verify that Activity Stream content can be saved and shared 1. Access the "Share" menu
2. Save page to pocket, share page trough most common services and system share services
3. Manage services
Manual Eng Team
6 Awesomebar Verify the changes implemented on the Awesomebar for the Activity Stream add-on TBD Manual -
7 Disable / Re-enable Activity Stream add-on Verify that the add-on can be disabled and enabled without any issues 1. Disable Activity Stream add-on
2. Verify that the browser UI returned to default functionality on the affected areas
3. Verify all saved content is kept after add-on disabling
4. Re-enable Activity Stream add-on and verify that the browser has changed back to the Activity Stream UI
5. Verify that all saved content is kept after re-enabling
Manual Eng Team
8 Uninstall Activity Stream add-on Verify that the UI changes are reverted and browser returns to default 1. Uninstall the Activity Stream add-on
2. Verify that the browser UI returned to default functionality on the affected areas
3. Verify that all saved content is kept after the uninstall
Manual Eng Team


Since Activity Stream is an add-on, you can install it from the following links:

Or you can install it trough Test Pilot experiments page. Note that the add-on only works on Firefox versions 45.x and up.

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project 02/15/2016 -
Study documentation/specs received from developers 02/15/2016 02/16/2016
QA - Test plan creation 01/25/2016 02/17/2016
QA - Test cases/Env preparation 02/18/2016 03/02/2016
QA - Nightly Testing 03/03/2016 -
QA - Aurora Testing
QA - Beta Testing
Release in Test Pilot Date 05/10/2016 -
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs management Bugzilla / GitHub (mainly)



  • Track the dates and build number where feature was released to Nightly
  • Track the dates and build number where feature was merged to Aurora
  • Track the dates and build number where feature was merged to Release/Beta

Risk analysis

Risk areas Mitigation
External dependencies on Test Pilot - We should do an end-to-end test of how we will integrate with the system.
- We should determine what our exact requirements are for sample size.
- Look into Telemetry as a contingency plan.
4 pieces of new UI/ concurrent development with Remote New Tab seems like a lot of work -We will push sharing/awesomebar to a future cycle.
- We will scale back development on Remote New Tab.
Reviews: data practices, AMO, legal - We should be covered by Test Pilot legal agreement so no additional legal review will be needed for MVP; we will not need to get AMO reviews, internal team code reviews will be enough for MVP launch.
Scope - Targeting only en-US?
- Accessibility requirements == best effort?




  • Summary of testing scenarios

Test Areas

Test Areas Covered Details
Private Window Yes
Multi-Process Enabled Yes
Multi-process Disabled Yes
Theme (high contrast) Yes
Mouse-only operation Yes
Keyboard-only operation Yes Navigation, selection and link opening only.
Display (HiDPI) Yes Testing on 4k resolution monitor.
Interraction (scroll, zoom) Yes
Usable with a screen reader No e.g. with NVDA
Usability and/or discoverability testing  ? Is this feature user friendly
Help/support interface required TBD Make sure link to support/help page exist and is easy reachable.
Support documents planned(written) TBD Make sure support documents are written and are correct.
Feature upgrades/downgrades data as expected Yes
Does sync work across upgrades Yes Tested if data is kept after upgrade or after import / sync profile data.
Requires install testing Yes The add-on can be installed trough Test Pilot or trough ".xpi" file.
Affects first-run or onboarding Yes The add-on requires some fetching time if the profile used has stored history and bookmarks.
Does this affect partner builds? Partner build testing yes/no options, add comment with details about who will lead testing
Enterprise Raise up the topic to developers to see if they are expecting to work different on ESR builds
Enterprise administration
Network proxies/autoconfig
ESR behavior changes
Locked preferences
Data Monitoring
Temporary or permanent telemetry monitoring Yes Testing was not conducted by SV QA Eng team.
Telemetry correctness testing Yes Testing was not conducted by SV QA Eng team.
Server integration testing Yes Testing was not conducted by SV QA Eng team.
Offline and server failure testing  ?
Load testing Yes Testing was not conducted by SV QA Eng team.
Add-ons If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on.
Addon API required?  ?
Comprehensive API testing  ?
Permissions  ?
Testing with existing/popular addons Yes Tested with addblock add-on.
Security Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature.
3rd-party security review  ?
Privilege escalation testing  ?
Fuzzing  ?
Web Compatibility depends on the feature
Testing against target sites No
Survey of many sites for compatibility No
Interoperability depends on the feature
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS  ?
Interaction of this feature with other browser features Yes

Test suite

Bug Work

Sign off


Check list

  • All test cases should be executed
  • Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


Nightly testing

Merge to Aurora Sign-off List of OSes that will be covered by testing

  • Link for the tests run -
    • Full Test suite -


Exit Criteria Status Notes/Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup No
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [IN PROGRESS]
Smoke Tests Execution [IN PROGRESS]
Automation Coverage
Performance Testing
All Defects Logged
Critical/Blockers Fixed and Verified
Daily Status Report (email/etherpad statuses/ gdoc with results)
Metrics/Telemetry N/A
QA Signoff - Nightly Release Email to be sent
QA Aurora - Full Testing
QA Signoff - Aurora Release Email to be sent
QA Beta - Full Testing
QA Signoff - Beta Release Email to be sent