Auto-tools/Projects/Firefox UI Tests

From MozillaWiki
Jump to: navigation, search

About

The Firefox UI tests are a collection of integration tests mainly for testing the user interface of Firefox. The tests and its harness are based on the Marionette testing framework, and enhances it for Firefox Desktop specific behavior, e.g. it adds a page object model aka Firefox-Puppeteer.

Team

This project is maintained by the Automation Crew, and the primary contact is Henrik Skupin.

Goals

The goal of this project is to increase the test coverage for new features in Firefox Desktop. Especially for those features which cannot be covered by any other existing test suite. To achieve that we will also have to continuously improve our harness and firefox-puppeteer page object model.

We want that test development will become as easy as writing manual tests. Therefore we only want to make use of synchronous test steps, and don't force the user to understand the DOM of the browsers interface.

Dependencies / Who will use this

For every feature enhancement which cannot be directly done in our own harness, we have to work together with the developers for Marionette.

The users will be:

  • Firefox QE

Contribution

We welcome contributions to Firefox UI Tests! We track issues inside the Firefox UI Tests component on Bugzilla, see below for a list of open bugs.

If you have a question, jump on #automation on irc.mozilla.org and look for whimboo, or maja_zf.

Source and docs

More information about the tests and how to run those can be found in the Mozilla Developer Network.

Active and Planned Projects

The currently most important projects for the firefox-ui-tests are the following:

  • Bug 1272236 - Let Mozmill-ci execute functional and update tests for Nightly builds on Linux64 via TaskCluster
  • Bug 1272228 - Make Firefox-ui-tests results from TaskCluster Tier-1 or Tier-2 on Treeherder
  • Bug 1272145 - Move Firefox-ui-tests from testing/firefox-ui-tests/tests to individual browser/toolkit components

Bug Table

ID Priority Component Summary Assigned to
1272589 -- Firefox UI Tests Intermittent test_toolbars.py TestLocationBar.test_load_url | TimeoutException: Timed out after 5.7 seconds Henrik Skupin (:whimboo)
1143020 -- Firefox UI Tests Convert dialog ui module to Marionette John Dorlus [:Silne30]
1329263 -- Firefox UI Tests Puppeteer throws ValueError when opening a new tab John Dorlus [:Silne30]
1036934 -- Firefox UI Tests Add test for content editing keyboard shortcuts
1036937 -- Firefox UI Tests Add test for history keyboard shortcuts
1088570 -- Firefox UI Tests Add automated tests for "Browse http/https and ftp websites with different settings"
1093599 -- Firefox UI Tests Add automated test for customized toolbar after restart
1113192 -- Firefox UI Tests Add automated test to check that prefs for openH264 plugin are correctly generated
1121691 -- Firefox UI Tests Implement module for handling observer notifications
1121698 -- Firefox UI Tests Implement fail-safe handling of chrome windows
1121702 -- Firefox UI Tests Implement module for handling events
1121705 -- Firefox UI Tests Implement fail-safe handling of tabs
1121710 -- Firefox UI Tests Fix MenuBar class for correct handling of menus
1121712 -- Firefox UI Tests Update readme for repository usage
1121725 -- Firefox UI Tests Port tabs library from mozmill-tests repository
1127786 -- Firefox UI Tests Refactor Puppeteer class for also having access to in UI classes
1129090 -- Firefox UI Tests Convert the tabbedbrowsing Mozmill tests to Marionette
1129092 -- Firefox UI Tests Convert the awesomebar Mozmill tests to Marionette
1131383 -- Firefox UI Tests Convert Mozmill test 'functional/testToolbar/testHomeButton.js' to Marionette
1131385 -- Firefox UI Tests Convert Mozmill test 'functional/testToolbar/testBackForwardButtons.js' to Marionette
1136640 -- Firefox UI Tests Add test for various video formats
1136641 -- Firefox UI Tests Add automated test for playing various audio formats
1136684 -- Firefox UI Tests Add automated tests for session restore after crash
1139544 -- Firefox UI Tests Add class for handling notification bars
1140014 -- Firefox UI Tests Documentation should explicitly state that a recent Nightly is required to run tests
1140291 -- Firefox UI Tests Create unit tests for Firefox UI harness
1140470 -- Firefox UI Tests Add equivalent of mozmill's waitForPageLoad to Firefox UI Tests libraries
1143121 -- Firefox UI Tests close method for popups throws NoSuchElementException in tearDown
1143928 -- Firefox UI Tests Refactor Deck and Panel classes to reduce code duplication
1146441 -- Firefox UI Tests Add signature checks for app bundle on OS X (maybe all once support across all platforms has been landed)
1148101 -- Firefox UI Tests Create a mozmill test for tracking protection
1168727 -- Firefox UI Tests TimeoutException for waiting for auto-complete popup in test_access_locationbar.py
1176422 -- Firefox UI Tests AutocompleteResults.get_matching_text can return non-matching text for results not of type bookmark
1176469 -- Firefox UI Tests Create a Firefox UI test for entering permanent PB mode
1177457 -- Firefox UI Tests "Failure during execution of the update test" should not be shown for functional and remote tests
1201186 -- Firefox UI Tests Verify Customize button on New Tab Page.
1206003 -- Firefox UI Tests Intermittent failure: test_safe_browsing_notification.py TestSafeBrowsingNotificationBar.test_notification_bar | NoSuchElementException: Unable to locate element: ignoreWarningButton
1207042 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_fallback_update.py TestFallbackUpdate.test_update | NoSuchWindowException: NoSuchWindowException: Unable to locate window: [1|3]
1207500 -- Firefox UI Tests Intermittent test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Update has been successfully downloaded
1214191 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_windows.py TestBaseWindow.test_open_close | TimeoutException: Timed out after 5.1 seconds with message: No focused window has been found.
1214198 -- Firefox UI Tests TEST-UNEXPECTED-FAIL | test_toolbars.py TestAutoCompleteResults.test_matching_text | AssertionError: u'a' not found in 'moz'
1231938 -- Firefox UI Tests Intermittent test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Update has been downloaded (transfer error: Failed) (2152398849)
1234175 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_windows.py TestBaseWindow.test_switch_to_and_focus | TimeoutException: Timed out after 5.0 seconds with message: Focus has not been set to chrome window handle "3"
1237766 -- Firefox UI Tests Write a Marionette test that downloads malware and verifies that it's blocked.
1239158 -- Firefox UI Tests Write a Marionette test that checks that visiting a website with a tracker in Private Browsing blocks the tracker
1257467 -- Firefox UI Tests Integrate Perfherder support for Firefox UI Tests
1258194 -- Firefox UI Tests Add tests for download history across restarts
1258664 -- Firefox UI Tests Add more details about which chrome window handles are leaked in tearDown of a FirefoxTestCase
1268087 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | MarionetteException: win.document.documentElement is null
1271809 -- Firefox UI Tests Move Firefox Puppeteer tests alongside the package and create a separate job in Treeherder
1272591 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_ssl_status_after_restart.py TestSSLStatusAfterRestart.test_ssl_status_after_restart | TimeoutException: TimeoutException: Timed out after 5.0 seconds with message: Identity popup has not been opened
1273498 -- Firefox UI Tests Update Puppeteer classes for MetaClass
1273600 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_unknown_issuer.py TestUnknownIssuer.test_unknown_issuer | NoSuchElementException: NoSuchElementException: Unable to locate element: errorCode
1274652 -- Firefox UI Tests Update tests shouldn't work on copies of the application
1275675 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_untrusted_connection_error_page.py TestUntrustedConnectionErrorPage.test_untrusted_connection_error_page | NoSuchElementException: Unable to locate element: returnButton
1276583 -- Firefox UI Tests TEST-UNEXPECTED-FAIL | test_windows.py TestBaseWindow.test_switch_to_and_focus | AssertionError: False is not true
1277485 -- Firefox UI Tests TEST-UNEXPECTED-FAIL | test_places.py TestPlaces.test_history | AssertionError: Lists differ: [u'http://127.0.0.1:39017/layo... != ['http://127.0.0.1:39017/layou...
1277935 -- Firefox UI Tests TEST-UNEXPECTED-ERROR | test_places.py TestPlaces.test_bookmarks | ScriptTimeoutException: ScriptTimeoutException: Timed out
1284119 -- Firefox UI Tests Intermittent test_places.py TestPlaces.test_plugins | ScriptTimeoutException: ScriptTimeoutException: Timed out
1285340 -- Firefox UI Tests TEST-UNEXPECTED-FAIL | test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: u'20160706030233' != '20160707083343' (target buildids different)
1285721 -- Firefox UI Tests Intermittent TEST-UNEXPECTED-ERROR | test_places.py TestPlaces.test_history | ScriptTimeoutException: ScriptTimeoutException: Timed out
1289115 -- Firefox UI Tests Fix timeout issues in Places.remove_all_history()
1289344 -- Firefox UI Tests Intermittent test_ev_certificate.py TestEVCertificate.test_ev_certificate | AssertionError: u'verifiedDomain' != 'verifiedIdentity'
1291948 -- Firefox UI Tests Intermittent test_end for test_unknown_issuer.py TestUnknownIssuer.test_unknown_issuer logged while not in progress. Logged with data:
1294353 -- Firefox UI Tests Intermittent test_suggest_bookmarks.py TestStarInAutocomplete.test_star_in_autocomplete | TimeoutException: Timed out after 10.3 seconds
1298233 -- Firefox UI Tests Intermittent test_appinfo.py TestAppInfo.test_valid_properties | AssertionError: u'20160825215758' != '20160825223519'
1304572 -- Firefox UI Tests Intermittent test_utils.py TestSanitize.test_sanitize_history | AssertionError: Lists differ: [u'http://127.0.0.1:44231/layo... != ['http://127.0.0.1:44231/layou...
1311987 -- Firefox UI Tests Intermittent test_direct_update.py TestDirectUpdate.test_update | IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Connection timed out after 60s)
1312359 -- Firefox UI Tests [tracking bug] Port Firefox UI functional tests to plain Marionette tests to reduce maintenance and complexity
1314456 -- Firefox UI Tests some windows update tests are ignoring config file versions and failing
1321432 -- Firefox UI Tests Intermittent test_safe_browsing_notification.py TestSafeBrowsingNotificationBar.test_notification_bar | TimeoutException: Timed out after 30.0 seconds with message: Expected target element "#main-feature" has not been found
1325047 -- Firefox UI Tests Test test_ssl_status_after_restart.py TestSSLStatusAfterRestart.test_ssl_status_after_restart fails in e10s mode
1337623 -- Firefox UI Tests Intermittent test_safe_browsing_warning_pages.py TestSafeBrowsingWarningPages.test_warning_pages | UnknownException: Reached error page: about:neterror?e=nssFailure2&u=https%3A//support.mozilla.org/t5/Protect-your-privacy/How-does-built-in-Phishing-and-
1355026 -- Firefox UI Tests Update Firefox UI update tests for new simplified update UI
1362308 -- Firefox UI Tests Intermittent test_security.py TestSecurity.test_get_address_from_certificate | UnknownException: Reached error page: about:neterror?e=netTimeout&u=https%3A//ssl-ev.mozqa.com (response taking too long)
1367319 -- Firefox UI Tests Check for updates has to take error panel into account
1372018 -- Firefox UI Tests Intermittent firefox-ui-functional PROCESS-CRASH | marionette.py | application crashed [None]
1372036 -- Firefox UI Tests Intermittent test_submit_unencrypted_info_warning.py TestSubmitUnencryptedInfoWarning.test_submit_unencrypted_info_warning | UnknownException: Reached error page: about:neterror?e=netTimeout&u=https%3A//ssl-dv.mozqa.com/data/firefox/security/unencryptedse
1373572 -- Firefox UI Tests Intermittent firefox-ui-update PROCESS-CRASH | marionette.py | application crashed [None]
1382154 -- Firefox UI Tests Intermittent test_restore_windows_after_restart_and_quit.py TestSessionStoreDisabled.test_restore_with_restart | AssertionError: Non private windows and tabs should have been restored.
1382453 -- Firefox UI Tests Intermittent test_restore_windows_after_restart_and_quit.py TestSessionStoreEnabled.test_with_variety | AssertionError: Non private windows and tabs should have been restored.
1298803 -- Firefox UI Tests Add test to quit Firefox via a shortcut Sanyam Khurana [:CuriousLearner]
1136237 -- Firefox UI Tests Find and use a better way to verify buttons' appearance in test_unknown_issuer.py Vaibhav Bhosale
1086334 P1 Firefox UI Tests Improve security tests for specific SSL/TLS protocol versions
1132629 P1 Firefox UI Tests Convert Mozmill functional and remote security tests to Marionette
1132680 P1 Firefox UI Tests Convert Mozmill test 'remote/testSecurity/testMD5HashSignature.js' to Marionette
1177417 P1 Firefox UI Tests Lots of failures in security tests due to UI changes of the identity popup
1129665 P2 Firefox UI Tests Convert Mozmill test 'functional/testTabbedBrowsing/testOpenInBackground.js' to Marionette Barbara Miller (:galgeek)
1129260 P2 Firefox UI Tests Convert Mozmill testCheckItemHighlight to Firefox UI Test Kamil Jozwiak [:kjozwiak]
790538 P2 Firefox UI Tests Enhance update tests to check for the 'what's new page' when `actions='silent'` given in update snippet
1129118 P2 Firefox UI Tests Convert Mozmill js test testSwitchToTab.js to a Firefox UI Test
1129634 P2 Firefox UI Tests Convert Mozmill test 'functional/testTabbedBrowsing/testNewTab.js' to Marionette
1129646 P2 Firefox UI Tests Convert Mozmill test 'functional/testTabbedBrowsing/testCloseTab.js' to Marionette
1129658 P2 Firefox UI Tests Convert Mozmill test 'functional/testTabbedBrowsing/testOpenInForeground.js' to Marionette
1129683 P2 Firefox UI Tests Convert Mozmill test 'functional/testTabbedBrowsing/testBackgroundTabScrolling.js' to Marionette
1130460 P2 Firefox UI Tests Convert Mozmill js test testGoButton.js to a Firefox UI Test
1131367 P2 Firefox UI Tests Convert Mozmill test 'functional/testAwesomeBar/testSuggestHistory.js' to Marionette
1131377 P2 Firefox UI Tests Convert Mozmill test 'functional/testAwesomeBar/testVisibleItemsMax.js' to Marionette
1138946 P2 Firefox UI Tests Backup and restore config files for software update tests
1387970 P2 Firefox UI Tests Permafail pkg_resources.DistributionNotFound: requests for Firefox UI tests
1201789 P3 Firefox UI Tests Add a test for shutting down Fx during a WebRTC call Nils Ohlmeier [:drno]
1384859 P5 Firefox UI Tests Intermittent test_restore_windows_after_restart_and_quit.py TestSessionStoreDisabled.test_no_restore_with_quit | IOError: Process has been unexpectedly closed (Exit code: 572) (Reason: [Errno 10061] No connection could be made because the
1396265 P5 Firefox UI Tests Intermittent test_restore_windows_after_restart_and_quit.py TestSessionStoreEnabled.test_with_variety | TimeoutException: Timed out after 5.1 seconds with message: No new tab has been opened.
1396803 P5 Firefox UI Tests Intermittent test_tabbar_session_restore_button.py TestNoTabbarSessionRestoreButton.test_pref_off_button_does_not_show | application crashed [@ mozilla::(anonymous namespace)::RunWatchdog(void*)]
1398741 P5 Firefox UI Tests [tier-3] Permafail - ERROR - The following files failed: 'macosx64-minidump_stackwalk'
1399761 P5 Firefox UI Tests Intermittent test_restore_windows_after_restart_and_quit.py TestSessionStoreDisabled.test_no_restore_with_quit | TimeoutException: Timed out after 5.0 seconds with message: No new tab has been opened.
1400107 P5 Firefox UI Tests Intermittent test_restore_windows_after_restart_and_quit.py TestSessionStoreDisabled.test_restore_with_restart | TimeoutException: Timed out after 5.7 seconds with message: No new tab has been opened.
1400630 P5 Firefox UI Tests Intermittent test_l10n.py TestL10n.test_dtd_entity_content | IOError: Process killed because the connection to Marionette server is lost. Check gecko.log for errors (Reason: Connection timed out after 360.0s)
1400791 P5 Firefox UI Tests Intermittent test_fallback_update.py TestFallbackUpdate.test_update | application crashed [@ RunWatchdog]
1402408 P5 Firefox UI Tests Intermittent IOError: Process killed after 120s because no connection to Marionette server could be established. Check gecko.log for errors

110 Total; 110 Open (100%); 0 Resolved (0%); 0 Verified (0%);