Auto-tools/Projects/Firefox UI Tests

From MozillaWiki
Jump to: navigation, search

About

The Firefox UI tests are a collection of integration tests mainly for testing the user interface of Firefox. The tests and its harness are based on the Marionette testing framework, and enhances it for Firefox Desktop specific behavior, e.g. it adds a page object model aka Firefox-Puppeteer.

Team

This project is maintained by the Automation Crew, and the primary contact is Henrik Skupin.

Goals

The goal of this project is to increase the test coverage for new features in Firefox Desktop. Especially for those features which cannot be covered by any other existing test suite. To achieve that we will also have to continuously improve our harness and firefox-puppeteer page object model.

We want that test development will become as easy as writing manual tests. Therefore we only want to make use of synchronous test steps, and don't force the user to understand the DOM of the browsers interface.

Dependencies / Who will use this

For every feature enhancement which cannot be directly done in our own harness, we have to work together with the developers for Marionette.

The users will be:

  • Firefox QE

Contribution

We welcome contributions to Firefox UI Tests! We track issues inside the Firefox UI Tests component on Bugzilla, see below for a list of open bugs.

If you have a question, jump on #automation on irc.mozilla.org and look for whimboo, or maja_zf.

Source and docs

More information about the tests and how to run those can be found in the Mozilla Developer Network.

Active and Planned Projects

The currently most important projects for the firefox-ui-tests are the following:

  • Bug 1272236 - Let Mozmill-ci execute functional and update tests for Nightly builds on Linux64 via TaskCluster
  • Bug 1272228 - Make Firefox-ui-tests results from TaskCluster Tier-1 or Tier-2 on Treeherder
  • Bug 1272145 - Move Firefox-ui-tests from testing/firefox-ui-tests/tests to individual browser/toolkit components

Bug Table

Full Query
ID Priority Component Summary Assigned to
1143020 -- Firefox UI Tests Convert dialog ui module to Marionette John Dorlus [:Silne30]
1036934 -- Firefox UI Tests Add test for content editing keyboard shortcuts
1036937 -- Firefox UI Tests Add test for history keyboard shortcuts
1088570 -- Firefox UI Tests Add automated tests for "Browse http/https and ftp websites with different settings"
1093599 -- Firefox UI Tests Add automated test for customized toolbar after restart
1113192 -- Firefox UI Tests Add automated test to check that prefs for openH264 plugin are correctly generated
1121691 -- Firefox UI Tests Implement module for handling observer notifications
1121698 -- Firefox UI Tests Implement fail-safe handling of chrome windows
1121702 -- Firefox UI Tests Implement module for handling events
1121710 -- Firefox UI Tests Fix MenuBar class for correct handling of menus
1121712 -- Firefox UI Tests Update readme for repository usage
1121725 -- Firefox UI Tests Port tabs library from mozmill-tests repository
1127786 -- Firefox UI Tests Refactor Puppeteer class for also having access to in UI classes
1129090 -- Firefox UI Tests Convert the tabbedbrowsing Mozmill tests to Marionette
1131383 -- Firefox UI Tests Convert Mozmill test 'functional/testToolbar/testHomeButton.js' to Marionette
1131385 -- Firefox UI Tests Convert Mozmill test 'functional/testToolbar/testBackForwardButtons.js' to Marionette
1136640 -- Firefox UI Tests Add test for various video formats
1136641 -- Firefox UI Tests Add automated test for playing various audio formats
1136684 -- Firefox UI Tests Add automated tests for session restore after crash
1139544 -- Firefox UI Tests Add class for handling notification bars
1140014 -- Firefox UI Tests Documentation should explicitly state that a recent Nightly is required to run tests
1140291 -- Firefox UI Tests Create unit tests for Firefox UI harness
1140470 -- Firefox UI Tests Add equivalent of mozmill's waitForPageLoad to Firefox UI Tests libraries
1143121 -- Firefox UI Tests close method for popups throws NoSuchElementException in tearDown
1146441 -- Firefox UI Tests Add signature checks for app bundle on OS X (maybe all once support across all platforms has been landed)
1148101 -- Firefox UI Tests Create a mozmill test for tracking protection
1168727 -- Firefox UI Tests TimeoutException for waiting for auto-complete popup in test_access_locationbar.py
1176422 -- Firefox UI Tests AutocompleteResults.get_matching_text can return non-matching text for results not of type bookmark
1176469 -- Firefox UI Tests Create a Firefox UI test for entering permanent PB mode
1177457 -- Firefox UI Tests "Failure during execution of the update test" should not be shown for functional and remote tests
1201186 -- Firefox UI Tests Verify Customize button on New Tab Page.
1206003 -- Firefox UI Tests Intermittent failure: test_safe_browsing_notification.py TestSafeBrowsingNotificationBar.test_notification_bar | NoSuchElementException: Unable to locate element: seeDetails
1207500 -- Firefox UI Tests Intermittent test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Update has been successfully downloaded
1214198 -- Firefox UI Tests TEST-UNEXPECTED-FAIL | test_toolbars.py TestAutoCompleteResults.test_matching_text | AssertionError: u'a' not found in 'moz'
1231938 -- Firefox UI Tests Intermittent test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Update has been downloaded (transfer error: Failed) (2152398849)
1234175 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_windows.py TestBaseWindow.test_switch_to_and_focus | TimeoutException: Timed out after 5.0 seconds with message: Focus has not been set to chrome window handle "3"
1237766 -- Firefox UI Tests Write a Marionette test that downloads malware and verifies that it's blocked.
1239158 -- Firefox UI Tests Write a Marionette test that checks that visiting a website with a tracker in Private Browsing blocks the tracker
1257467 -- Firefox UI Tests Integrate Perfherder support for Firefox UI Tests
1258194 -- Firefox UI Tests Add tests for download history across restarts
1258664 -- Firefox UI Tests Add more details about which chrome window handles are leaked in tearDown of a FirefoxTestCase
1271809 -- Firefox UI Tests Move Firefox Puppeteer tests alongside the package and create a separate job in Treeherder
1272591 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_ssl_status_after_restart.py TestSSLStatusAfterRestart.test_ssl_status_after_restart | TimeoutException: TimeoutException: Timed out after 5.0 seconds with message: Identity popup has not been opened
1273498 -- Firefox UI Tests Update Puppeteer classes for MetaClass
1273600 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_unknown_issuer.py TestUnknownIssuer.test_unknown_issuer | NoSuchElementException: NoSuchElementException: Unable to locate element: errorCode
1274652 -- Firefox UI Tests Update tests shouldn't work on copies of the application
1275675 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_untrusted_connection_error_page.py TestUntrustedConnectionErrorPage.test_untrusted_connection_error_page | NoSuchElementException: Unable to locate element: returnButton
1276583 -- Firefox UI Tests TEST-UNEXPECTED-FAIL | test_windows.py TestBaseWindow.test_switch_to_and_focus | AssertionError: False is not true
1284119 -- Firefox UI Tests Intermittent test_places.py TestPlaces.test_plugins | ScriptTimeoutException: ScriptTimeoutException: Timed out
1289344 -- Firefox UI Tests Intermittent test_ev_certificate.py TestEVCertificate.test_ev_certificate | AssertionError: u'verifiedDomain' != 'verifiedIdentity'
1291948 -- Firefox UI Tests Intermittent test_end for test_unknown_issuer.py TestUnknownIssuer.test_unknown_issuer logged while not in progress. Logged with data:
1312632 -- Firefox UI Tests Intermittent test_mixed_script_content_blocking.py TestMixedScriptContentBlocking.test_mixed_content_page | TimeoutException: Timed out after 5.0 seconds with message: Insecure {plugin|script one|stylesheet} has been unblocked
1321432 -- Firefox UI Tests Intermittent test_safe_browsing_notification.py TestSafeBrowsingNotificationBar.test_notification_bar | TimeoutException: Timed out after 300.0 seconds with message: Expected target element "#main-feature" has not been found
1362308 -- Firefox UI Tests Intermittent test_security.py TestSecurity.test_get_address_from_certificate | UnknownException: Reached error page: about:neterror?e=netTimeout&u=https%3A//ssl-ev.mozqa.com (response taking too long)
1372020 -- Firefox UI Tests Intermittent test_security_notification.py TestSecurityNotification.test_invalid_cert | AssertionError: 'expired.badssl.com' not found in u''
1372036 -- Firefox UI Tests Intermittent test_submit_unencrypted_info_warning.py TestSubmitUnencryptedInfoWarning.test_submit_unencrypted_info_warning | UnknownException: Reached error page: about:neterror?e=netTimeout&u=https%3A//ssl-dv.mozqa.com/data/firefox/security/unencryptedse
1373561 -- Firefox UI Tests Intermittent test_about_private_browsing.py TestAboutPrivateBrowsing.testCheckAboutPrivateBrowsing | NoSuchElementException: Unable to locate element: learnMore
1382453 -- Firefox UI Tests Intermittent test_restore_windows_after_restart_and_quit.py TestSessionStoreEnabled.test_with_variety | AssertionError: Non private windows and tabs should have been restored.
1382494 -- Firefox UI Tests Intermittent test_dv_certificate.py TestDVCertificate.test_dv_cert | UnknownException: Reached error page: about:neterror?e=netTimeout&u=https%3A//mozilla-intermediate.badssl.com
1445862 -- Firefox UI Tests Intermittent firefox UI Aborting task - max run time exceeded!
1511312 -- Firefox UI Tests Fix update tests to set "app.update.disabledForTesting" to False
1136237 -- Firefox UI Tests Find and use a better way to verify buttons' appearance in test_unknown_issuer.py Vaibhav Bhosale
1129665 P2 Firefox UI Tests Convert Mozmill test 'functional/testTabbedBrowsing/testOpenInBackground.js' to Marionette Barbara Miller (:galgeek)
790538 P2 Firefox UI Tests Enhance update tests to check for the 'what's new page' when `actions='silent'` given in update snippet
1129634 P2 Firefox UI Tests Convert Mozmill test 'functional/testTabbedBrowsing/testNewTab.js' to Marionette
1129646 P2 Firefox UI Tests Convert Mozmill test 'functional/testTabbedBrowsing/testCloseTab.js' to Marionette
1129658 P2 Firefox UI Tests Convert Mozmill test 'functional/testTabbedBrowsing/testOpenInForeground.js' to Marionette
1129683 P2 Firefox UI Tests Convert Mozmill test 'functional/testTabbedBrowsing/testBackgroundTabScrolling.js' to Marionette
1214191 P2 Firefox UI Tests TEST-UNEXPECTED-ERROR | test_windows.py TestBaseWindow.test_open_close | TimeoutException: Timed out after 5.1 seconds with message: No focused window has been found.
1201789 P3 Firefox UI Tests Add a test for shutting down Fx during a WebRTC call
1312359 P3 Firefox UI Tests [tracking bug] Port Firefox UI functional tests to plain Marionette tests to reduce maintenance and complexity
1325047 P3 Firefox UI Tests Test test_ssl_status_after_restart.py TestSSLStatusAfterRestart.test_ssl_status_after_restart fails in e10s mode
1405591 P5 Firefox UI Tests Add automated test for the Youtube website Ioana Crisan
1406399 P5 Firefox UI Tests Add automated test for the Facebook website johansonbugs
1277935 P5 Firefox UI Tests TEST-UNEXPECTED-ERROR | testing/firefox-ui/tests/puppeteer/test_places.py TestPlaces.test_bookmarks | ScriptTimeoutException: ScriptTimeoutException: Timed out
1285721 P5 Firefox UI Tests Intermittent TEST-UNEXPECTED-ERROR | test_places.py TestPlaces.test_history | ScriptTimeoutException: ScriptTimeoutException: Timed out
1289115 P5 Firefox UI Tests Fix timeout issues in Places.remove_all_history()
1400780 P5 Firefox UI Tests Intermittent test_restore_windows_after_restart_and_quit.py TestSessionStoreDisabled.test_restore_with_restart | AssertionError: Non private browsing windows should have
1400791 P5 Firefox UI Tests Intermittent test_fallback_update.py TestFallbackUpdate.test_update | application crashed [@ RunWatchdog]
1416759 P5 Firefox UI Tests [meta] Add automated tests for Web Compatibility functionality
1459910 P5 Firefox UI Tests Intermittent testing/firefox-ui/tests/functional/security/test_ssl_disabled_error_page.py TestSSLDisabledErrorPage.test_ssl_disabled_error_page | AssertionError: u'The connection has timed out' != u'Secure Connection Failed'
1479227 P5 Firefox UI Tests Intermittent testing/firefox-ui/tests/functional/security/test_security_notification.py TestSecurityNotification.test_secure_website | TimeoutException: Timeout loading page after 300000ms
1479228 P5 Firefox UI Tests Intermittent testing/firefox-ui/tests/puppeteer/test_security.py TestSecurity.test_get_address_from_certificate | TimeoutException: Timeout loading page after 300000ms
1479229 P5 Firefox UI Tests Intermittent testing/firefox-ui/tests/puppeteer/test_security.py TestSecurity.test_get_address_from_certificate | UnknownException: Reached error page: about:neterror?e=netTimeout&u=https%3A//extended-validation.badssl.com/&c=UTF-8&f=regular&d=The%20serve
1480794 P5 Firefox UI Tests Intermittent testing/firefox-ui/tests/functional/security/test_mixed_content_page.py TestMixedContentPage.test_mixed_content | TimeoutException: Timeout loading page after 300000ms
1490999 P5 Firefox UI Tests Intermittent testing/firefox-ui/tests/functional/safebrowsing/test_initial_download.py TestSafeBrowsingInitialDownload.test_safe_browsing_initial_download | AssertionError: 'mozstd-trackwhite-digest256.pset' not found in [u'google4', ...]
1491284 P5 Firefox UI Tests Intermittent testing/firefox-ui/tests/functional/security/test_ssl_disabled_error_page.py TestSSLDisabledErrorPage.test_ssl_disabled_error_page | TimeoutException: Timed out after 5.0 seconds with message: Click on the restore button didn't trigger a page
1511169 P5 Firefox UI Tests Intermittent testing\firefox-ui\tests\functional\sessionstore\test_restore_windows_after_windows_shutdown.py TestWindowsShutdownRegisterRestart.test_manual_restart | AssertionError: Non private browsing windows should have
1511288 P5 Firefox UI Tests Intermittent testing\firefox-ui\tests\functional\sessionstore\test_restore_windows_after_windows_shutdown.py TestWindowsShutdown.test_with_variety | AssertionError: Non private browsing windows should have
1512602 P5 Firefox UI Tests Intermittent testing/firefox-ui/tests/functional/safebrowsing/test_notification.py TestSafeBrowsingNotificationBar.test_notification_bar | UnknownException: Reached error page: about:neterror?e=networkProtocolError&u=https%3A//www.itisatrap.org/f
1513288 P5 Firefox UI Tests Intermittent testing\firefox-ui\tests\functional\sessionstore\test_restore_windows_after_restart_and_quit.py TestSessionStoreEnabledAllWindows.test_with_variety | AssertionError: Non private browsing windows should have
1298803 P5 Firefox UI Tests Add test to quit Firefox via a shortcut Sanyam Khurana [:CuriousLearner]

92 Total; 92 Open (100%); 0 Resolved (0%); 0 Verified (0%);