Web Apps integration/TestPlan

From MozillaWiki
Jump to navigation Jump to search

Web Apps Integration

Feature Release Target Dev Lead QA Lead Dev Status QA Status Health
Web Apps Integration Firefox 13 Felipe Gomes Jason Smith In Progress Planning At risk

Feature Summary

This feature focuses on being able to find web applications to install on Firefox through the home tab and installing and uninstalling these applications natively different operating systems. These installable web applications are essentially websites built in web technologies (e.g. HTML, CSS, JavaScript) that users can interact with through a chromeless shell in a native environment. For a website to become a web application, a developer creates an app manifest for their website and hosts it on an origin (e.g. www.yourhost.com) where the website is located. Then, the app developer adds this manifest to a store (e.g. Firefox store) to allow the app to be installed to users' machines.

Users can then go to this store to install this application to their native machine. Note that installable applications can be paid or free. For paid applications, users will have to pay for the application through Paypal. Upon making the application payment, a user's machine receives a receipt to prove that the application is paid for. Then a user can install this paid application to their machine on Win XP+ or Mac OS X. See the native installation flow to understand how native applications are installed a user's machine on Windows XP+ and Mac OS X. After a user selects to install an application, a confirmation should appear asking the user to confirm installation of the application to a particular file location on the user's machine. Then, upon confirmation, the application should be installed to the user's machine. At this point, the user should understand where the application was installed on their machine.

After applications are installed, users can run these applications in their native environment in a chromeless shell both online and offline. If the application is paid, validation will need to take place using the receipt for the application to ensure that the user paid for the application. On windows, applications are typically ran as shortcuts from the desktop, but may also be ran from the start menu in executables. On Mac OS X, applications are ran through the ~/Applications directory typically. When an application is launched, a chromeless window starts using Firefox's webapp mode under the hood with the web application running in the shell. In the task manager in Windows, you would see that the application is process running on the system under the name of the web application. Within the shell, users can take actions within the application, such as logging in, clicking links within the origin of the application, playing a game, and more. Within the shell itself, users have a menu to allow basic application actions such as quitting, cut/copy, and more. When a user is done using the application, they can quit using the application.

If the user no longer wants the application installed on their machine, they can uninstall the application from their machine. For windows, uninstalling can take place either directly in the directory location of the application or through add and remove programs. On Mac OS X, uninstalling occurs by moving the application to trash and deleting the application from the trash bin. Upon uninstalling an application, all locally stored app data that previously created during installation should be cleared.

Testing Scope

Major Features

The scope of testing of this feature focuses on the following major features:

  • Home tab web apps integration
  • Installing native applications
  • Launching native applications
  • Using native applications
  • Uninstalling native applications

Edge Cases

The following special edge cases need to be taken into account when testing the major parts of this feature:

  • Browser crashes
  • Loss of internet connection (e.g. offline mode)
  • App state changes (i.e. free to paid, paid to free)
  • Invalid or no receipts for paid apps
  • Local vs. roaming windows profiles
  • System restore

Testing Strategy

The following testing approaches shall be used to assess the scope of testing stated above:

Development of Manual Test Cases: Manual test cases shall be created to test the major portions of features in the testing scope. Each test case shall specify release quality requirement specifying which test cases need to pass within Firefox Aurora, Beta, or Release respectively.

Exploratory Testing of Tier 1 Apps: Exploratory testing of using tier 1 native applications shall be conducted to ensure that our highest priority applications are supported on our platform. When exploratory testing these applications, a checklist shall be used to test major features we would expect from these applications such as account management, migrating off the origin of the application, and more. Upon finishing exploratory testing of these applications, bugs shall be logged for functional issues and new manual and scenario tier 1 app test cases shall be developed. Additionally, if problems with the app itself are found, then evangelism bugs shall be filed for those issues.

Development of Value-Added Automation: Automation shall be developed for manual test cases that carry the highest value add in comparison to the overhead to the build the automation. To determine the value added versus the overhead, an approach for automation shall be documented next to each manual test case to determine if the payoff to build the automation exceeds the overhead cost. Technologies that shall be used will likely be either Mochitest or Mozmill, but this could change based on what manual test cases exist for this feature.

Tier 1 App Scenario Tests: Scenario tests shall be used on this desktop feature to simulate real-use of the application. Examples being considered for scenario tests follow the model shown at the bottom of the MWC Pod Demo Script. Similar to the manual test cases, each test case shall specify a release quality requirement (e.g. Aurora, Beta, Release).

Sign Off Criteria

Aurora

  • No bugs found with a severity level of major or higher in aurora test cases
  • Aurora-level functional test cases pass
    • Windows XP+ and Mac OS X 10.5+ OS-specific aurora test cases pass
  • Aurora-level tier 1 app scenario test cases pass
  • No unresolved bugs with a severity level of major or higher that cause aurora-level test cases to fail

Beta

  • Tier 1 app scenario aurora and beta test cases all pass
  • Aurora and beta level functional test cases pass
    • Windows XP+ and Mac OS X 10.5+ OS-specific aurora and beta test cases pass
  • No unresolved bugs with a severity level of major or higher that cause aurora or beta test cases to fail
  • No regressions in Firefox linked to the feature's code changes detected with a severity of major or higher

Release

  • No bugs found with a severity level of major or higher in test cases
  • All major functional test cases pass
    • Windows XP+ and Mac OS X 10.5+ OS-specific test cases pass
  • Tier 1 app scenario test cases pass
  • No unresolved bugs with a severity level of major or higher that cause test cases to fail

Infrastructure Requirements

<List any infrastructure requirements that are needed to test this feature>

Test Cases and Results

Test Cases

  • Manual test cases and automation approach can be found in the Spreadsheet: Google Spreadsheet
  • Exploratory test cases of tier 1 apps can be found in the Spreadsheet: <insert link here>
  • Tier 1 app scenario tests can be found in the Spreadsheet: <insert link here>

Test Results

  • Manual test case results shall be tracked in this spreadsheet: <insert link here>
  • Exploratory test results of tier 1 apps shall be tracked in this spreadsheet: <insert link here>
  • Tier 1 app scenario test results shall be tracked in this spreadsheet: <insert link here>

Important Bugs

Full Query
ID Summary Priority Status
725533 webapps.js looks leaky -- RESOLVED
732280 Intermittent icon failures on Windows XP when installing an app natively -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Important References

Open Questions

Question 1

Install Use Case - Windows 7

"User confirms app installation and app is installed into program files and start menu"

jsmith:

Talking with Tim, we were under the impression that when apps are installed, they are installed to %APPDATA% on windows under a directory that models the origin of the application. Is this %APPDATA% installation correct? Checking this since as shown above, I'm seeing a mention about installing to the program files in windows, which I thought was not the place we were installing applications.

timA:

It's infeasible to install to Program Files. %APPDATA% or %LOCALAPPDATA% are good options (Google Chrome, for example, installs to %LOCALAPPDATA%).

I believe that %APPDATA% and %LOCALAPPDATA% have the following pros and cons for large organizations that utilize roaming profiles, but this has not been investigated:

  • %APPDATA%: Installed apps will magically appear at any terminal that a user logs into. If Firefox is installed on that machine, the apps should magically work. Otherwise they will pop up error dialogs indicating that Firefox is required.
  • %LOCALAPPDATA%: Users will have to install their apps at each terminal they log into.

Question 2

Re-launch already opened app - Windows 7/Mac OS X

  • From Start Menu (or task bar or desktop shortcut) click app
  • App window is refocused

jsmith:

Checking with Tim, I think this is different behavior than what is currently implemented on Windows. Right now, on windows, if say, try to start favimon when it is already running, another instance of favimon starts up. On Mac OS X, I believe it only allows one app to be running at one time. What is the expected behavior in this scenario?

timA:

From the discussion in #openwebapps, I think we want to allow only one instance of each app to run at a time (in line with what the spec says).

jsmith:

Okay. I also recall Dan mentioning something about the idea of having a single instance of the process running, but having multiple windows of the app. That idea falls in line with how Firefox operates - for example, if I click a desktop shortcut on Windows 7, a new window starts up for Firefox. The overarching question I ask is - How can a user have multiple windows of the same native application running? Would the approach Firefox uses be right to use in-line with apps (single process, multiple windows)? The scenario I'm thinking about where a need for multiple windows of an app is something like this:

Concern Type: Productivity

Application: Basecamp

Window 1: Workspace 1 for team X, to-do list

Window 2: Workspace 2 for team Y, to-do list

View on a User's System: Dual Monitor - Workspace 1 on screen 1, workspace 2 on screen 2

Question 3

(jsmith) Outside of the free app expected behavior, what additionally happens with paid apps? For example, where does the receipt of the purchase get stored?

Question 4

(jsmith) How is app reinstall handled?

Question 5

(jsmith) What is the expected behavior when a paid app is reinstalled that is different than just installing a paid app?

Question 6

(jsmith) What's the difference with a different account when reinstalling the same paid app?

Retrieved from "https://wiki.mozilla.org/index.php?title=Web_Apps_integration/TestPlan&oldid=406806"