QA/BrowserID Automation

How It Works

Jenkins is the virtual host that runs the cmds that are sent to saucelabs which has webdriver running. So any scripts/run-all.js logs are on jenkins and webdriver logs are on saucelabs. Capice'? KICK OFF RUN https://ci.mozilla.org/

• login with ldap passwd
• jenkins kicks off runs based on project names (e.g. identity.browserid.dev.mac.firefox.14)
• if you 'configure' inside the project, you'll see the cmd line args etc...
• if you want to kick something off manually, get on IRC and PM jenkins with something like this:

jenkins: build identity.browserid.dev.mac.firefox.14

• NOTE: the 'build now' on the jenkin's UI doesn't work

RESULTS

• How this works: Jenkins looks at results/test.xml file with pass/fail which is generated when you specify output at --output xunit
• Run is complete - look at last fail or last success permalinks or in build history list
• Click on a failed run
• If there isn't any inteligent error reporting. Click on Console link
• find the failing test, and goto the saucelabs URL to view the webdriver output
• NOTE: sign in to saucelabs with your env variable user:passwd

https://saucelabs.com

Selenium

Known Issues

Safari: browser cache is not cleared between each test (except chrome, FF) http://code.google.com/p/selenium/issues/detail?id=40

Opera: goes fullscreen on mac opens in new tab instead of specific sized pop up (mac only)

Android: only supports stock browser with no chrome ( can't user Firefox or Chrome )

iOS: dunno must provision hooked up to xcode to install app

some browsers can't close alert dialog (safari, opera) selenium.common.exceptions.WebDriverException: Message: u"A modal dialog was opened. The SafariDriver does not support interacting with modal dialogs. To avoid hanging your test, the alert has been dismissed. For more information, see http://code.google.com/p/selenium/issues/detail?id=3862\nBuild info: version: '2.31.0', revision: '1bd294d', time: '2013-02-27 20:52:59'\nSystem info: os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.8.2', java.version: '1.6.0_41'\nSession ID: null\nDriver info: org.openqa.selenium.safari.SafariDriver\nCapabilities [{platform=MAC, javascriptEnabled=true, cssSelectorsEnabled=true, secureSsl=true, browserName=safari, takesScreenshot=true, version=6.0.2}]" ; Screenshot: available via screen ; Stacktrace: Method newInstance0 threw an error in NativeConstructorAccessorImpl.java

Jenkins and Saucelabs

Feature Requests

• need a way to embed saucelabs URL in results.xml to reduce hunting for logs
• link to github test to see src

General

• General build failure from fetching modules:
  • npm http GET https://github.com/downloads/lloyd/node-statsd/0509f85.tgz
  • npm ERR! browserid@1.0.0-b1 preinstall: `node ./scripts/lockdown`

I'm going to log a bug on this

• User email validation
  • This seems flakey, often you'll see 'timeout' error as the browser sits on .watch() for validation
• Errors are intermittent and all over the map for some browsers

• • osx_safari_5 example:

run 322

• change-password-test.js.12: back to the first browser: should be signed out of 123done on reload: succeeds
• health-check-tests.js.12: switch back to main window, look for the email in acct mgr, then log out: succeeds
• new-user-secondary-test.js.11: verify logged in automatically to 123done in first browser: succeeds

run 320

• primary-shuts-down-single-email.js.10: Authed user tries to log in: succeeds

run 319

• health-check-tests.js.12: switch back to main window, look for the email in acct mgr, then log out: succeeds

Issues

Approach: need to get all builds to show green on Jenkins dashboard. Starting with are more stable platforms, here's a breakdown of why we have failures with Jenkins/Saucelabs.

• dev.win7_chrome
  • primary-shuts-down-single-email - testidp.org doesn't get confirmed
• dev.osx_chrome_14
  • webdriver went stale and didn't kill existing browser
• dev.linux_firefox_16
  • sign in button just spins

Saucelabs

• Currently using out of date browsers and configs. Need to update.