Web Apps integration/TestPlan
- 1 Overview
- 2 Feature Summary
- 3 Testing Scope
- 4 Testing Strategy
- 5 Sign Off Criteria
- 6 Infrastructure Requirements
- 7 Test Case Management
- 8 Automation
- 9 Bugs
- 10 Important References
- 11 Open Questions
|Feature||Release Target||Dev Lead||QA Lead||Dev Status||QA Status||Health|
|Web Apps Integration||Firefox 15||Felipe Gomes||Jason Smith||Landed||In Progress||OK|
Users can then go to this store to install this application to their native machine. Note that installable applications can be paid or free. For paid applications, users will have to pay for the application through Paypal. Upon making the application payment, a user's machine receives a receipt to prove that the application is paid for. Then a user can install this paid application to their machine on Win XP+ or Mac OS X. See the native installation flow to understand how native applications are installed a user's machine on Windows XP+ and Mac OS X. After a user selects to install an application, a confirmation should appear asking the user to confirm installation of the application to a particular file location on the user's machine. Then, upon confirmation, the application should be installed to the user's machine. At this point, the user should understand where the application was installed on their machine.
After applications are installed, users can run these applications in their native environment in a chromeless shell both online and offline. If the application is paid, validation will need to take place using the receipt for the application to ensure that the user paid for the application. On windows, applications are typically ran as shortcuts from the desktop, but may also be ran from the start menu in executables. On Mac OS X, applications are ran through the /Applications directory typically. When an application is launched, a chromeless window starts using Firefox's webapp mode under the hood with the web application running in the shell. In the task manager in Windows, you would see that the application is process running on the system under the name of the web application. Within the shell, users can take actions within the application, such as logging in, clicking links within the origin of the application, playing a game, and more. Within the shell itself, users have a menu to allow basic application actions such as quitting, cut/copy, and more. When a user is done using the application, they can quit using the application.
If the user no longer wants the application installed on their machine, they can uninstall the application from their machine. For windows, uninstalling can take place either directly in the directory location of the application or through add and remove programs. On Mac OS X, uninstalling occurs by moving the application to trash and deleting the application from the trash bin. Upon uninstalling an application, all locally stored app data that previously created during installation should be cleared.
The scope of testing of this feature focuses on the following major features:
- Home tab web apps integration
- Installing native applications
- Launching native applications
- Using native applications
- Uninstalling native applications
- Browser crashes
- Loss of internet connection (e.g. offline mode)
- App state changes (i.e. free to paid, paid to free)
- Invalid or no receipts for paid apps
- Local vs. roaming windows profiles
- System restore
The following testing approaches shall be used to assess the scope of testing stated above:
Development of Manual Test Cases: Manual test cases shall be created to test the major portions of features in the testing scope. Each test case shall specify a quality requirement (smoke or feature signoff). Applicable sets of these test cases (smoke and feature signoff) should be ran each beta release that are value-added to run to capture bugs.
Exploratory Testing of Marketplace Top Apps: Exploratory testing of using top native applications shall be conducted to ensure that our highest priority applications are supported on our platform. When exploratory testing these applications, a checklist below will help guide the direction of exploratory testing:
- Does the application avoid using engine specific content?
- Does the application show out of origin content?
- Does clicking out of origin links that are not authentication go to the browser?
- Does application pop-ups and platform integration work as expected?
- Does flash and experimental features work as expected?
For problems discovered that involves a top app, a whiteboard marker of '[topapps]' shall be specified. Any problems with the app itself shall have evangelism bugs logged if the app is a top app.
Development of Value-Added Automation: Automation shall be developed for smoke test cases that carry the highest value add in comparison to the overhead to the build the automation. See the automation section for more details.
Sign Off Criteria
- All Desktop WebRT blockers are fixed
- Smoke tests should pass on Win XP, Vista, 7 (32/64 bit), OS X 10.5 - 10.7 with no major issues
- No significant regressions in Desktop Firefox (e.g. crashes due to this implementation)
- Flagged polish issues are fixed
- Smoke and important feature signoff tests should pass on Win XP, Vista, 7 (32/64 bit), OS X 10.5 - 10.7 with no major issues
- No significant regressions in Firefox due to this implementation
- Same as Beta
mozqa.com's firefox test cases section shall need to be turned into a web app to allow for testing the underlying platform when using apps with desktop. If new test cases for platform are necessary, then they will need to be uploaded as test cases to the mozqa.com test server.
For testing the manifests with the apps lifecycle, testmanifest.com shall be used to do this. Manifests shall be attached to test cases. Then, the tester shall have to copy and paste the manifest into an applicable testmanifest.com URL, save it, and install the application. From there, other test cases can be done as needed for the apps lifecycle (e.g. uninstall application).
Test Case Management
Test cases for this feature shall be tracked in MozTrap. In MozTrap, they will be flagged as a Desktop Firefox 15 feature in the product version tag. Additionally, smoke tests will be flagged with the "smoke" tag. Feature signoff test cases will be flagged with the "feature signoff" tag. Each of these test cases are broken up into categories such as functional, security, resilience, localization, and performance and are tagged as such.
Note: Ideas for test cases are documented here
Automation for this feature shall intend to target smoke tests that touch the apps lifecycle (e.g. install, uninstall) and the use of the application (e.g. can I open a pop-up?). Since add-ons are not intended to be supported in the web runtime, this automation then shall be built using Marionette. This framework is necessary to use as Marionette does not rely on add-ons to run, where as other frameworks such as Mozmill do rely on add-ons to operate correctly. In addition to Marionette, we will need to build a python library that allows common operating system interaction in OS X and Windows to analyze major parts of the apps lifecycle (e.g. was a webapp.json file created, was the webapprt exe file created). For launching this automation, the following process has been suggested:
- mozrunner (mozprocess) launch the process
- marionette communicates with the process, tests
- (there is also the python-side of the tests)
- mozrunner/mozprocess terminates process
- (as a backup, the CI system kills processes and reboots the system - might require improving mozrunner for this need)
First steps to getting this setup will require ensuring that marionette can be built with firefox, as this has not been tested yet. Upon completing the desktop integration with marionette, we will need to write a python library for common OS interactions. Upon completion on the python library, we will then need to build a proof of concept automated test utilizing marionette with the python library. This test should target the basic apps lifecycle steps (e.g. install app, launch app, use app, uninstall app). Following completion of the proof of concept, further automated tests can be continued to be built as needed to get automated coverage of smoke tests with desktop web apps.
A summary of the current bugs in the system are found here. The below lists summarizes the bugs that need to be verified:
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
- Feature wiki page - Specification for the Web Apps Integration feature
- App Directory - Sample apps to install for testing
- Apps Wiki - The apps team wiki page
- Apps Overview - The vision of the apps project
- Apps WebRT - The web runtime specification
- Apps Developer Docs - The apps developer documentation
- About Me Web App - Sample Web App
- Flickr Connector Web App - Example web app
- Photosite Connector Web App - Example web app
- Marketplace Beta Test Run Results - Results from Test Run for Marketplace Beta Signoff
Reinstallation of Paid Apps with a Different Account Question
(jsmith) What's the difference with a different account when reinstalling the same paid app?
Install App on Two Different Firefox Profiles
(jsmith) What is the expected behavior if I install an app on two different firefox profiles? Will one app overwrite the other entirely? Partly?