Apps/QA/OWA Extension

OWA Extension

QA Lead:	Mohamed Dabbagh
Lead Developer:	Mike Hanson

Quick Start Guide

Here is a quick start guide to help you dive right in to Open Web Apps:

1. Download and install the extension - Latest built can be found here
2. Install apps - Sample apps can be found here
3. View the Dashboard - Apps can be found by clicking the 'Apps' button at the bottom right of your browser or by going here

It's that simple! Once you have apps installed, you can launch them by clicking on them in the Dashboard. You can also go to 'about:builds' (type this without the quotes in the URL bar) and launch apps there, view the manifest or delete apps.

Join us on IRC:

• #openwebapps - everything openwebapps
• #automation - testing/automation (not specifically openwebapps)
• #qa - testing (not specifically openwebapps)
  

Testing Initiative

Focus

• Extension (installing, deleting and launching apps)
• Dashboard
• App Sync
• Web Activities/Intents (not Q4)
  

The testing of the OWA extension will focus on the extension itself (for example, installation/deletion/compatibility and the mozApps.* api), on the dashboard (for example, adding/removing apps), on App Sync and on web activities. The idea is to determine the requirements/specifications of the extension and create manual test cases that will test the basics of the extension/dashboard. Initially, the test cases will be subject to a great deal of changes depending on the requirements and will need to be kept up to date. This applies to the automation as well where keeping the automated test cases in sync with their corresponding manual test cases is essential. Further testing can be done using exploratory methods or a more advanced set of test cases. A proper way of getting apps that can be used for testing will be needed.

The basic iteration of the testing process will hopefully be as follows:

1. Tester will install the latest build of the OWA extension
   
2. Tester will verify any bugs that have been resolved on the latest extension build
3. Tester will run automation tests to make sure no regressions have occured
4. Tester will test using manual test cases and exploratory testing as well -- Once automation is in place, the tester can run the automation for the basic functionality testing
5. Tester will file new bugs as necessary
6. Tester will create new test cases as necessary and make sure the current test cases are up to date
   

NOTE: The extension will be available and will be needed at launch but eventually it will be built into the browser. Therefore, some of the test cases developed might not be needed anymore when this occurs.

Test Cases

A set of test cases is being created that focuses on manual testing strategies. The main functionality document being used to aid in the creation of the test cases can be found here. The document is split up into spreadsheets relating to a function/area to test. If there are any questions, comments or additions (other test cases you think are valuable but are not in the document) please feel free to contact David Clarke or Mohamed Dabbagh (irc: mdabbagh).

The test cases document is actively being updated and can be found here.

There are quite a few questions that should be answered in order to have a good understanding of what the functionality of the extension and apps will be like. Answering these questions is crucial in having a solid set of manual test cases. The list of questions is being tracked here. Please feel free to update the document and add any other questions or answer some of the questions there already.

Native App Generation

Native app generation just means allowing the user to install apps natively on their machine. This is equivalent to installing a program or application. If you would like to check it out, please follow these steps (advanced) or simply install one of the extensions from http://people.mozilla.com/~mdabbagh/openwebapps/extension/ that have been built on or after November 10, 2011:

1. Install the addon-sdk (from github as the release version will not work) - https://github.com/mozilla/addon-sdk
   
2. Download the oauthorizer package into the packages folder of the addon-sdk - https://github.com/mozilla/oauthorizer
   
3. Clone the openwebapps project - https://github.com/mozilla/openwebapps
   
4. In a terminal, cd into the addon-sdk folder and type: bin\activate (windows) or source bin/activate (linux)
   
5. cd into openwebapps/addons/jetpack
   
6. Type: cfx run - Note: Currently, using a release version of FF that is less than FF9 will cause an error relating to the icon. Therefore, either use FF9 (currently in beta) or use nightly or aurora by including the --binary command and passing in the path to firefox (.exe for windows) nightly/aurora (ex: cfx --binary="c:\Program Files\Nightly\firefox.exe" run).
   

Test Accounts

This will have test accounts that can be used for testing Open Web Apps.

Extension Builds

The latest builds for the OWA extension can be found here. The builds are created whenever a commit is made to the openwebapps github repo. The format is openwebapps-<commit id>-<date>.xpi. For automation, openwebapps.xpi within the 'latest' folder is used to avoid making changes to the addon.ini file for every test run. To build the extension yourself from the github repository, please do the following:

1. git clone http://github.com/mozilla/openwebapps
2. cd openwebapps
   
3. make xpi (this will build the xpi file and place it in your current directory)

You can drag the xpi file to your Firefox window to install it. You can also host the file on the web and the user would just click on the xpi file to install it to their Firefox.

Bug Tracking/Reporting

This section will contain a list of all the bugs that we have filed or are tracking. Bugs are being logged in the Web Apps product in Bugzilla. Below is a quick link to the product to file a bug. There is also a tracking bug for the extension that is being used to track the progress of work and what needs to be done for an alpha release.

• Extension Tracking Bug
  
• Web Apps Product - Bugzilla

Bug Reporting Format:

Of course there are multiple ways of reporting a bug and what information to include or format to use can vary, this is just one way that is currently being used. Once you find a bug relating to Apps, you can do the following:

• Test the issue on all supported platforms
• Choose the right component under the Web Apps product in Bugzilla
• Choose the proper severity
• Select which OS the problem occurs for
• If you know who the issue should be assigned to, select the person from the 'Assignee' field. However, this is best left to the triage team if you are not sure.
• Enter a summary of the problem - Be specific and include all relevant information. For example, "Apps do not launch" is not a very effective summary. However, saying 'Apps do not launch from the Dashboard without having the apps extension installed" is a much better summary
• In the Description field:
  • Enter a very detailed description of the problem and include any relevant information or observations
  • Enter very detailed 'Steps to Reproduce' - This should be easy to follow and detailed enough that any person could reproduce the issue exactly as stated
    • If possible, you should attach the xpi file you used to reproduce the problem. Alternately, you can say the following: 'Extension can be found at http://people.mozilla.com/~dclarke/openwebapps/Extension/ and look for <xpi file you used that contains the commit ID and the date>'
  • Optionally, you could enter 'Alternate Steps to Reproduce' to show the different ways the problem occurs
  • Enter the 'Actual Result' - Enter what happens right after the user performs the last step in the steps to reproduce
  • Enter the 'Expected Result'
• Add any attachments as necessary
  • This could include screenshots (highly recommended), videos or any files that you think are useful/helpful

Automation

The automation efforts will revolve around automating the manual test cases using mozmill. Since currently there is not much to work with, a basic skeleton framework is being looked into that can hopefully hold common API's that will be used when automating tthe test cases.

A demo test run was created with MozMill that installs an app, runs the app, closes the app and then deletes it. The patch for the demo test can be found here. Henrik Skupin is going to write a blog post about OWA automation and will include a demo that runs this script.

Test Server

An already existing nodejs server was modified to host test apps that we can use for load and smoke testing. To use the test server, please follow these steps (in linux):

Setup:

1. git clone git://github.com/mozilla/openwebapps.git
   
2. cd openwebapps
   
3. Type: 'git submodule init' and then 'git submodule update'
   
4. Make sure to have node installed - If not, you can get it from here. You can also follow one of the methods here.

Run:

1. cd site/tests/
2. Type: 'node run.js' - All necessary servers will start
   
3. Launch the URL after 'Testing server started, to run tests go to:' and replace index.html with appstore.html
   
4. A webpage will be shown that contains all the test apps

How It Works:

It's just a nodejs server that creates a separate server for each app that we have and hosts the necessary files within each server. The directory where the apps are stored is in site/tests/apps and each app has a folder within this directory.If you would like to add apps, all you have to do is add a folder within the apps directory and place the necessary files within that folder (please look at repo under site/tests/apps/ for examples). Please note the following:

• The manifest file must be called: manifest.webapp (this should be changed within the code to be more dynamic)
• If there is a problem with the page formatting, it's most likely because the icon pic used for the app is not compatible. The icon pic must be 128x128px in order to show up properly in the test server and in the Dashboard. You can use software such as gimp to adjust the pic size.
  

Documents and Links

QA Related

Test Cases Document

OWA Extension/Apps Questions

OWA Extension Builds

Extension Tracking Bug

Marketplace Tracking Bug

General Tracked Bugs

Firefox Store Apps Staging Server

Information/Tracking

Dashboard Spec Document

Apps Sync Spec

Apps Wiki

Apps Overview - Main

WebRT - Useful information (can be used to create more test cases)

Apps Github Repository

Web Activities/Intents

Explore

Sample Apps (for demo)

Firefox Store Staging Server

Tasks

The following is the current list of tasks being worked on. Please feel free to take any task that is not already claimed or if you would like to help, please contact the person under the 'Assigned' column.

Task Description	Status	Comments/Notes	Assigned	Priority
Determining specifications/requirements for extension and apps to use as reference for manual test cases	Completed	All the available documentation that is being used to track the progress the of the extension and to aid in creating test cases can be found below under Documents/Links.	mdabbagh	P2
Determining where/how to get apps to use for testing. Building a server that hosts dummy apps.	Completed	A nodejs server has been created and integrated with the nodejs server code from the openwebapps repo. The page is similar to the sample apps page and allows us to control the apps that are uploaded. The code has been reviewed and merged with the main branch. It seems like we are leaning more towards using the staging server (found below under Documents/Links) for end to end testing and our created server for stress and smoke testing.	dclarke, mdabbagh	P1
Creating a manual test cases plan for the extension, dashboard and apps.	In Progress	The first run through has been completed. Further test cases are being developed and missing information is being tracked down. If you know where we can get more documents/explanation of functionality to help us in completing the test cases, please feel free to add it to the list of documents below and/or contact us. Questions have arised from writing the test cases and they can be found in the links section below.	mdabbagh	P2
Creating a place to get OWA extension builds from	Completed	A people.mozilla account has been created where OWA extension builds can be downloaded/installed from. Please see 'Builds' above for more information and for the link to the page. This is only a temporary solution as a better solution is to use the Add-on builder, which is currently being fixed.	mdabbagh	P1
Write a proof of concept test for OWA extension/app testing	In Review	A mozmill test has been created that will display a proof of concept automation test just for demo purposes. The test can be found here. Uploaded another patch with the requested changes. All the problems being experienced deal with the extension build. Please read the comments in the bug.	mdabbagh	P3
Start testing the extension, Dashboard and apps	In Progress	Starting to do some basic testing and to file some bugs. Currently testing App Sync. Bugs are/have been filed in Bugzilla.	mdabbagh	P1
Determining how to track changes and issues related to the extension/apps/dashboard	Completed	Searching Bugzilla for open issues related to the extension/apps/dashboard. A tracking bug for the extension has been found and another tracking bug for all of apps has been found as well. There is also a general Bugzilla search that contains bugs relating to the extension/apps/Dashboard. Please look under Bug Tracking above.	mdabbagh	P3
Start automation efforts for the staging server and for our test server	In Progress	Starting to plan out the automation strategy. Determining how to setup the automation and what API's are needed to support creating automated tests.	mdabbagh	P1

Infrastructure Requirements

Alternate Build Generation and Hosting

• A people.mozilla.com account has been created where the xpi file for the OWA extension will be placed as needed: mdabbagh@people.mozilla.com
  
• Developers or testers can make builds and place them there (for example, a bug is fixed and a quick check is needed using the developer's local code)
• Allows customization of builds

Getting Started:

To get started and to upload xpi files, you first need a people.mozilla.com account. Please follow this link in order to setup the account and to learn how to upload/download files.

If you only want to download the xpi files, please go here. If you would like an xpi file (extension) built, please send an email to mdabbagh@mozilla.com or ping mdabbagh in IRC (#qa, #openwebapps, #automation). You can also build the file yourself by following the instructions here (please use linux, Ubuntu for example, instead of Windows. It's much easier).

Marketplace instance managed by Al, limited access.